Shading should be a last resort: https://jlbp.dev/JLBP-18.html
It tends to cause more problems than it solves. At best it's a stopgap measure when you don't have the resources to fix the real problem. In this case it sounds like the real issue is that AVRO is not stable. There are at least three other solutions in a case like this: 1. Fix Avro at the root. 2. Fork Avro and then fix it. 3. Stop depending on Avro. None of these are trivial which is why shading gets considered. However shading doesn't fix the underlying problems, and ultimately makes a product as unreliable as its least reliable dependency. :-( On Thu, Jan 16, 2020 at 2:01 AM jincheng sun <sunjincheng...@gmail.com> wrote: > > I found that there are several dependencies shaded and planned to made as > vendored artifacts in [1]. I'm not sure why Avro is not shaded before. From > my point of view, it's a good idea to shade Avro and make it a vendored > artifact if there are no special reasons blocking us to do that. Regarding to > how to create a vendored artifact, you can refer to [2] for more details. > > Best, > Jincheng > > [1] https://issues.apache.org/jira/browse/BEAM-5819 > [2] https://github.com/apache/beam/blob/master/vendor/README.md > > > Tomo Suzuki <suzt...@google.com> 于2020年1月16日周四 下午1:18写道： >> >> I've been upgrading dependencies around gRPC. This Avro-problem is >> interesting to me. >> I'll study BEAM-8388 more tomorrow. >> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik <lc...@google.com> wrote: >> > >> > +Tomo Suzuki +jincheng sun >> > There have been a few contributors upgrading the dependencies and >> > validating things not breaking by running the majority of the post commit >> > integration tests and also using the linkage checker to show that we >> > aren't worse off with respect to our dependency tree. Reaching out to them >> > to help your is your best bet of getting these upgrades through. >> > >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon <atdi...@gmail.com> wrote: >> >> >> >> I meant to mention that we must use Avro 1.9.x as we rely on some schema >> >> resolution fixes not present in 1.8.x - so am indeed blocked. >> >> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon <atdi...@gmail.com> wrote: >> >>> >> >>> It looks like Avro version dependency from Beam has come up in the past >> >>> [1, 2]. >> >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my usage of >> >>> Avro 1.9.x. >> >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has >> >>> some dependencies on Avro classes only available in 1.8.x. >> >>> >> >>> Wondering if anyone else is similar blocked and what it would take to >> >>> prioritize Beam upgrading to 1.9.x or better using a shaded version so >> >>> that clients can use their own Avro version for their own coding >> >>> purposes. (Eg, I parse Avro messages from a KafkaIO source and need >> >>> 1.9.x for this but am perfectly happy if Beam's Avro coding facilities >> >>> used a shaded other version.) >> >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling >> >>> community for discussion. >> >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >> >>> [2] https://github.com/apache/beam/pull/9779 >> >>> >> >> >> -- >> Regards, >> Tomo -- Elliotte Rusty Harold elh...@ibiblio.org
