Re: Beam's Avro 1.8.x dependency
I'm not saying it is a best practice. I'm suggesting we should figure
out whether it's a best practice, worst practice, or somewhere in
between and document in which situations it is or is not a good idea.
Up till now I haven't really considered it an alternative to shading
so it seems worth thinking about more deeply.
On Fri, Jan 17, 2020 at 1:35 PM Kenneth Knowles wrote:
>
>
>
> On Fri, Jan 17, 2020 at 10:21 AM Elliotte Rusty Harold
> wrote:
>>
>> Thanks. That's a very clear explanation. I should think about how to
>> incorporate that into https://jlbp.dev
>
>
> Might want to give it some time before calling it a best practice. Maybe
> we'll come to regret it :-). But I'm willing to say it is a better
> alternative to shading in most cases, with the major exception being
> generated code (like proto libs) that depend on libraries (like
> protobuf-java). Ideally, proto compiler would vendor the dep into each
> generated lib so we don't have to deal with the issue at all. Instead we do
> it ourselves.
>
> Kenn
>
>>
>>
>>
>> On Fri, Jan 17, 2020 at 12:12 PM Aaron Dixon wrote:
>> >
>> > @Tomo Suzuki Thanks for looking at this and your thorough analysis// let
>> > me know if I can help with any regression testing, CRs, or more context
>> > anything else.
>> >
>> > ---
>> >
>> > @Elliotte I was a little confused at first re: vendoring vs shading but
>> > here is how I understand it (happy to be corrected if anything I say here
>> > is a mischaracterization); I tend to over-explain for my own clarity
>> > fwiw--using Guava as an example:
>> >
>> > With SHADING you have your project (e.g., Beam) depend directly on Guava
>> > and its public package names. Your build will then compile and link to
>> > these package names and only after-the-fact [1] rewrite the byte-code to
>> > rename the guava package. This entails both byte-code rewriting the Guava
>> > class files (to rename the packages of the Guava code itself) and
>> > byte-code rewriting all of your project class files that import Guava (to
>> > change the import to use the new package name).
>> >
>> > With VENDORING you release a renamed Guava ("vendor") JAR and depend on
>> > that directly from your project. So your project's source code itself
>> > imports the (repackated) vendor JAR and depends on the renamed packages
>> > directly. Your project build then has no need for shading or byte code
>> > rewriting of its own classes. I see vendoring making it very clear what is
>> > going on when looking at a project's source code and its dependencies, and
>> > there are likely other tangible brass-tacks benefits to vendoring besides
>> > the conceptual clarity over shading.
>> >
>> > [1] You can see that Maven shade:shade goal binds by default to Maven's
>> > package phase (which is well after after compilation).
>> >
>> > On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
>> >>
>> >> Ismaël and I think the short-term solution for Aaron's issue is to
>> >> define Beam's own class for Avro's TimestampConversion. Luckily it's
>> >> the only blocker for Avro 1.9 I found.
>> >> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
>> >> PR https://github.com/apache/beam/pull/10628 .
>> >>
>> >> Regards,
>> >> Tomo
>> >>
>> >> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
>> >> wrote:
>> >> >
>> >> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote:
>> >> > >
>> >> > > Regarding Google's advice about shading: don't go to a "one version
>> >> > > rule" monorepo for advice about solving diamond dependencies in the
>> >> > > wild.
>> >> >
>> >> > FWIW these docs came out of the open source, Github, multirepo, many
>> >> > versions part of Google Cloud, not the mono repo world of google3.
>> >> > Though I will say the more time I spend trying to deal with these
>> >> > issues the more I love the monorepo.
>> >> >
>> >> > > It is a useful description of the pitfalls. We (and Flink before us,
>> >> > > and likely many more) are already doing something that avoids many of
>> >> > > them or makes them less likely. Building a separate vendored library
>> >> > > is simpler and more robust than the "shade during build".
>> >> > >
>> >> >
>> >> > I've only encountered "vendoring" as distinct from shading in Beam.
>> >> > Are there more details about this anywhere? A quick Google search only
>> >> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
>> >> > doc that seemed to use it as a synonym for shading.
>> >> >
>> >> >
>> >> > --
>> >> > Elliotte Rusty Harold
>> >> > elh...@ibiblio.org
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Tomo
>>
>>
>>
>> --
>> Elliotte Rusty Harold
>> elh...@ibiblio.org
--
Elliotte Rusty Harold
elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
On Fri, Jan 17, 2020 at 10:21 AM Elliotte Rusty Harold
wrote:
> Thanks. That's a very clear explanation. I should think about how to
> incorporate that into https://jlbp.dev
Might want to give it some time before calling it a best practice. Maybe
we'll come to regret it :-). But I'm willing to say it is a better
alternative to shading in most cases, with the major exception being
generated code (like proto libs) that depend on libraries (like
protobuf-java). Ideally, proto compiler would vendor the dep into each
generated lib so we don't have to deal with the issue at all. Instead we do
it ourselves.
Kenn
>
>
> On Fri, Jan 17, 2020 at 12:12 PM Aaron Dixon wrote:
> >
> > @Tomo Suzuki Thanks for looking at this and your thorough analysis// let
> me know if I can help with any regression testing, CRs, or more context
> anything else.
> >
> > ---
> >
> > @Elliotte I was a little confused at first re: vendoring vs shading but
> here is how I understand it (happy to be corrected if anything I say here
> is a mischaracterization); I tend to over-explain for my own clarity
> fwiw--using Guava as an example:
> >
> > With SHADING you have your project (e.g., Beam) depend directly on Guava
> and its public package names. Your build will then compile and link to
> these package names and only after-the-fact [1] rewrite the byte-code to
> rename the guava package. This entails both byte-code rewriting the Guava
> class files (to rename the packages of the Guava code itself) and byte-code
> rewriting all of your project class files that import Guava (to change the
> import to use the new package name).
> >
> > With VENDORING you release a renamed Guava ("vendor") JAR and depend on
> that directly from your project. So your project's source code itself
> imports the (repackated) vendor JAR and depends on the renamed packages
> directly. Your project build then has no need for shading or byte code
> rewriting of its own classes. I see vendoring making it very clear what is
> going on when looking at a project's source code and its dependencies, and
> there are likely other tangible brass-tacks benefits to vendoring besides
> the conceptual clarity over shading.
> >
> > [1] You can see that Maven shade:shade goal binds by default to Maven's
> package phase (which is well after after compilation).
> >
> > On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
> >>
> >> Ismaël and I think the short-term solution for Aaron's issue is to
> >> define Beam's own class for Avro's TimestampConversion. Luckily it's
> >> the only blocker for Avro 1.9 I found.
> >> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
> >> PR https://github.com/apache/beam/pull/10628 .
> >>
> >> Regards,
> >> Tomo
> >>
> >> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
> >> wrote:
> >> >
> >> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles
> wrote:
> >> > >
> >> > > Regarding Google's advice about shading: don't go to a "one version
> rule" monorepo for advice about solving diamond dependencies in the wild.
> >> >
> >> > FWIW these docs came out of the open source, Github, multirepo, many
> >> > versions part of Google Cloud, not the mono repo world of google3.
> >> > Though I will say the more time I spend trying to deal with these
> >> > issues the more I love the monorepo.
> >> >
> >> > > It is a useful description of the pitfalls. We (and Flink before
> us, and likely many more) are already doing something that avoids many of
> them or makes them less likely. Building a separate vendored library is
> simpler and more robust than the "shade during build".
> >> > >
> >> >
> >> > I've only encountered "vendoring" as distinct from shading in Beam.
> >> > Are there more details about this anywhere? A quick Google search only
> >> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
> >> > doc that seemed to use it as a synonym for shading.
> >> >
> >> >
> >> > --
> >> > Elliotte Rusty Harold
> >> > elh...@ibiblio.org
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Tomo
>
>
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>
Re: Beam's Avro 1.8.x dependency
Thanks. That's a very clear explanation. I should think about how to
incorporate that into https://jlbp.dev
On Fri, Jan 17, 2020 at 12:12 PM Aaron Dixon wrote:
>
> @Tomo Suzuki Thanks for looking at this and your thorough analysis// let me
> know if I can help with any regression testing, CRs, or more context anything
> else.
>
> ---
>
> @Elliotte I was a little confused at first re: vendoring vs shading but here
> is how I understand it (happy to be corrected if anything I say here is a
> mischaracterization); I tend to over-explain for my own clarity fwiw--using
> Guava as an example:
>
> With SHADING you have your project (e.g., Beam) depend directly on Guava and
> its public package names. Your build will then compile and link to these
> package names and only after-the-fact [1] rewrite the byte-code to rename the
> guava package. This entails both byte-code rewriting the Guava class files
> (to rename the packages of the Guava code itself) and byte-code rewriting all
> of your project class files that import Guava (to change the import to use
> the new package name).
>
> With VENDORING you release a renamed Guava ("vendor") JAR and depend on that
> directly from your project. So your project's source code itself imports the
> (repackated) vendor JAR and depends on the renamed packages directly. Your
> project build then has no need for shading or byte code rewriting of its own
> classes. I see vendoring making it very clear what is going on when looking
> at a project's source code and its dependencies, and there are likely other
> tangible brass-tacks benefits to vendoring besides the conceptual clarity
> over shading.
>
> [1] You can see that Maven shade:shade goal binds by default to Maven's
> package phase (which is well after after compilation).
>
> On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
>>
>> Ismaël and I think the short-term solution for Aaron's issue is to
>> define Beam's own class for Avro's TimestampConversion. Luckily it's
>> the only blocker for Avro 1.9 I found.
>> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
>> PR https://github.com/apache/beam/pull/10628 .
>>
>> Regards,
>> Tomo
>>
>> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
>> wrote:
>> >
>> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote:
>> > >
>> > > Regarding Google's advice about shading: don't go to a "one version
>> > > rule" monorepo for advice about solving diamond dependencies in the wild.
>> >
>> > FWIW these docs came out of the open source, Github, multirepo, many
>> > versions part of Google Cloud, not the mono repo world of google3.
>> > Though I will say the more time I spend trying to deal with these
>> > issues the more I love the monorepo.
>> >
>> > > It is a useful description of the pitfalls. We (and Flink before us, and
>> > > likely many more) are already doing something that avoids many of them
>> > > or makes them less likely. Building a separate vendored library is
>> > > simpler and more robust than the "shade during build".
>> > >
>> >
>> > I've only encountered "vendoring" as distinct from shading in Beam.
>> > Are there more details about this anywhere? A quick Google search only
>> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
>> > doc that seemed to use it as a synonym for shading.
>> >
>> >
>> > --
>> > Elliotte Rusty Harold
>> > elh...@ibiblio.org
>>
>>
>>
>> --
>> Regards,
>> Tomo
--
Elliotte Rusty Harold
elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
On Fri, Jan 17, 2020 at 9:12 AM Aaron Dixon wrote:
> @Tomo Suzuki Thanks for looking at this and your thorough analysis// let
> me know if I can help with any regression testing, CRs, or more context
> anything else.
>
> ---
>
> @Elliotte I was a little confused at first re: vendoring vs shading but
> here is how I understand it (happy to be corrected if anything I say here
> is a mischaracterization); I tend to over-explain for my own clarity
> fwiw--using Guava as an example:
>
> With SHADING you have your project (e.g., Beam) depend directly on Guava
> and its public package names. Your build will then compile and link to
> these package names and only after-the-fact [1] rewrite the byte-code to
> rename the guava package. This entails both byte-code rewriting the Guava
> class files (to rename the packages of the Guava code itself) and byte-code
> rewriting all of your project class files that import Guava (to change the
> import to use the new package name).
>
> With VENDORING you release a renamed Guava ("vendor") JAR and depend on
> that directly from your project. So your project's source code itself
> imports the (repackated) vendor JAR and depends on the
> renamed packages directly. Your project build then has no need for shading
> or byte code rewriting of its own classes. I see vendoring making it very
> clear what is going on when looking at a project's source code and its
> dependencies, and there are likely other tangible brass-tacks benefits to
> vendoring besides the conceptual clarity over shading.
>
Exactly. Vendoring can also refer to just making your own copy of a
library. The conceptual clarity extends beyond humans into clarity for
tooling. IDEs understand it better, and it is much easier to manage builds.
And of course it avoids jar bloat and speeds builds. It isn't perfect. You
lose interop.
Kenn
> [1] You can see that Maven shade:shade goal binds by default to Maven's
> package phase (which is well after after compilation).
>
> On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
>
>> Ismaël and I think the short-term solution for Aaron's issue is to
>> define Beam's own class for Avro's TimestampConversion. Luckily it's
>> the only blocker for Avro 1.9 I found.
>> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
>> PR https://github.com/apache/beam/pull/10628 .
>>
>> Regards,
>> Tomo
>>
>> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
>> wrote:
>> >
>> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles
>> wrote:
>> > >
>> > > Regarding Google's advice about shading: don't go to a "one version
>> rule" monorepo for advice about solving diamond dependencies in the wild.
>> >
>> > FWIW these docs came out of the open source, Github, multirepo, many
>> > versions part of Google Cloud, not the mono repo world of google3.
>> > Though I will say the more time I spend trying to deal with these
>> > issues the more I love the monorepo.
>> >
>> > > It is a useful description of the pitfalls. We (and Flink before us,
>> and likely many more) are already doing something that avoids many of them
>> or makes them less likely. Building a separate vendored library is simpler
>> and more robust than the "shade during build".
>> > >
>> >
>> > I've only encountered "vendoring" as distinct from shading in Beam.
>> > Are there more details about this anywhere? A quick Google search only
>> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
>> > doc that seemed to use it as a synonym for shading.
>> >
>> >
>> > --
>> > Elliotte Rusty Harold
>> > elh...@ibiblio.org
>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>
Re: Beam's Avro 1.8.x dependency
@Tomo Suzuki Thanks for looking at this and your thorough analysis// let me
know if I can help with any regression testing, CRs, or more context
anything else.
---
@Elliotte I was a little confused at first re: vendoring vs shading but
here is how I understand it (happy to be corrected if anything I say here
is a mischaracterization); I tend to over-explain for my own clarity
fwiw--using Guava as an example:
With SHADING you have your project (e.g., Beam) depend directly on Guava
and its public package names. Your build will then compile and link to
these package names and only after-the-fact [1] rewrite the byte-code to
rename the guava package. This entails both byte-code rewriting the Guava
class files (to rename the packages of the Guava code itself) and byte-code
rewriting all of your project class files that import Guava (to change the
import to use the new package name).
With VENDORING you release a renamed Guava ("vendor") JAR and depend on
that directly from your project. So your project's source code itself
imports the (repackated) vendor JAR and depends on the
renamed packages directly. Your project build then has no need for shading
or byte code rewriting of its own classes. I see vendoring making it very
clear what is going on when looking at a project's source code and its
dependencies, and there are likely other tangible brass-tacks benefits to
vendoring besides the conceptual clarity over shading.
[1] You can see that Maven shade:shade goal binds by default to Maven's
package phase (which is well after after compilation).
On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
> Ismaël and I think the short-term solution for Aaron's issue is to
> define Beam's own class for Avro's TimestampConversion. Luckily it's
> the only blocker for Avro 1.9 I found.
> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
> PR https://github.com/apache/beam/pull/10628 .
>
> Regards,
> Tomo
>
> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
> wrote:
> >
> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote:
> > >
> > > Regarding Google's advice about shading: don't go to a "one version
> rule" monorepo for advice about solving diamond dependencies in the wild.
> >
> > FWIW these docs came out of the open source, Github, multirepo, many
> > versions part of Google Cloud, not the mono repo world of google3.
> > Though I will say the more time I spend trying to deal with these
> > issues the more I love the monorepo.
> >
> > > It is a useful description of the pitfalls. We (and Flink before us,
> and likely many more) are already doing something that avoids many of them
> or makes them less likely. Building a separate vendored library is simpler
> and more robust than the "shade during build".
> > >
> >
> > I've only encountered "vendoring" as distinct from shading in Beam.
> > Are there more details about this anywhere? A quick Google search only
> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
> > doc that seemed to use it as a synonym for shading.
> >
> >
> > --
> > Elliotte Rusty Harold
> > elh...@ibiblio.org
>
>
>
> --
> Regards,
> Tomo
>
Re: Beam's Avro 1.8.x dependency
Ismaël and I think the short-term solution for Aaron's issue is to define Beam's own class for Avro's TimestampConversion. Luckily it's the only blocker for Avro 1.9 I found. I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and PR https://github.com/apache/beam/pull/10628 . Regards, Tomo On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold wrote: > > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote: > > > > Regarding Google's advice about shading: don't go to a "one version rule" > > monorepo for advice about solving diamond dependencies in the wild. > > FWIW these docs came out of the open source, Github, multirepo, many > versions part of Google Cloud, not the mono repo world of google3. > Though I will say the more time I spend trying to deal with these > issues the more I love the monorepo. > > > It is a useful description of the pitfalls. We (and Flink before us, and > > likely many more) are already doing something that avoids many of them or > > makes them less likely. Building a separate vendored library is simpler and > > more robust than the "shade during build". > > > > I've only encountered "vendoring" as distinct from shading in Beam. > Are there more details about this anywhere? A quick Google search only > turned up one Beam doc that pointed to Go docs and a cloud endpoints > doc that seemed to use it as a synonym for shading. > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org -- Regards, Tomo
Re: Beam's Avro 1.8.x dependency
On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote: > > Regarding Google's advice about shading: don't go to a "one version rule" > monorepo for advice about solving diamond dependencies in the wild. FWIW these docs came out of the open source, Github, multirepo, many versions part of Google Cloud, not the mono repo world of google3. Though I will say the more time I spend trying to deal with these issues the more I love the monorepo. > It is a useful description of the pitfalls. We (and Flink before us, and > likely many more) are already doing something that avoids many of them or > makes them less likely. Building a separate vendored library is simpler and > more robust than the "shade during build". > I've only encountered "vendoring" as distinct from shading in Beam. Are there more details about this anywhere? A quick Google search only turned up one Beam doc that pointed to Go docs and a cloud endpoints doc that seemed to use it as a synonym for shading. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
Regarding Google's advice about shading: don't go to a "one version rule" monorepo for advice about solving diamond dependencies in the wild. It is a useful description of the pitfalls. We (and Flink before us, and likely many more) are already doing something that avoids many of them or makes them less likely. Building a separate vendored library is simpler and more robust than the "shade during build". Ismaël's point #2 is important: we can't shade or vendor Avro if we intend to use it with user-generated code. Generated code with external dependencies requires coordination with the vendoring, as we do for portability + gRPC. The ProtoCoder uses non-vendored proto for this reason. The one totally internal use of Avro I am aware of is BigQueryIO. This could perhaps use a vendored Avro. But OTOH it is already in an isolated module so it is less severe. And as many have pointed out, upgrading a dep across a breaking change is a breaking change. "Stop depending on Avro" is a breaking change as well. So if we are going to do that, moving it out of core is a more valuable breaking change. But perhaps highlighting Gleb's comment: we can build a separate library/artifacts for providing an AvroCoder that uses Avro 1.9.x (and potentially make a separate one for 1.8.x and encourage users to use that). We might be able to make Avro 1.8.x optional for the core SDK, finding a way for a user to pin to 1.9 as long as they don't touch the parts of the SDK that use 1.8. Kenn On Thu, Jan 16, 2020 at 1:49 PM Aaron Dixon wrote: > Looks like there's some strategy to get to the right solution here and > that it may likely involve breaking compatibility. > > One option for myself would be to strip the Beam JAR of AvroCoder and > combine with the old AvroCoder from Beam 2.16 -- this would allow me to > upgrade Beam but of course is rather hacky. > > On second thought, was the breaking change from Beam 2.16->2.17 really > necessary? If not, could AvroCoder be restored to a 1.9.x "compatible" > implementation and kept this way for the Beam 2.1x version lineage? > > This seems like a somewhat fair ask given the way that I'm suddenly > blocked --- however I do realize this is somewhat of a technicality; ie, > Beam 2.16-'s compatibility with my usage of Avro 1.9.x was incidental. > > But, still, if the changes to AvroCoder weren't necessary, restoring back > would unblock me and anyone else using Avro 1.9.x (surely I'm not the only > one!?) > > > On Thu, Jan 16, 2020 at 12:22 PM Elliotte Rusty Harold > wrote: > >> Avro does not follow semver. They update the major version when the >> serialization format changes and the minor version when the API >> changes in a backwards incompatible way. See >> https://issues.apache.org/jira/browse/AVRO-2687 >> >> On Thu, Jan 16, 2020 at 12:50 PM Luke Cwik wrote: >> > >> > Does avro not follow semantic versioning and upgrading to 1.9 should >> have been backwards compatible or does our usage reach into the internals >> of avro? >> > >> > On Thu, Jan 16, 2020 at 6:16 AM Ismaël Mejía wrote: >> >> >> >> I forgot to explain why the most obvious path (just upgrade Avro to >> version >> >> 1.9.x) is not a valid long term solution. Other systems Beam runs on >> top of >> >> (e.g. Spark!) also leak Avro into their core so in the moment Spark >> moves up >> >> to Avro 1.9.x Spark runner users will be in a really fragile position >> where >> >> things will work until they don't (similar to Aaron's case) so a >> stronger reason >> >> to getAvro out of Beam core. >> >> >> >> >> >> On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold < >> elh...@ibiblio.org> wrote: >> >>> >> >>> Shading should be a last resort: >> >>> >> >>> https://jlbp.dev/JLBP-18.html >> >>> >> >>> It tends to cause more problems than it solves. At best it's a stopgap >> >>> measure when you don't have the resources to fix the real problem. In >> >>> this case it sounds like the real issue is that AVRO is not stable. >> >>> There are at least three other solutions in a case like this: >> >>> >> >>> 1. Fix Avro at the root. >> >>> 2. Fork Avro and then fix it. >> >>> 3. Stop depending on Avro. >> >>> >> >>> None of these are trivial which is why shading gets considered. >> >>> However shading doesn't fix the underlying problems, and ultimately >> >>> makes a product as unreliable as its least reliable dependency. :-( >> >>> >> >>> On Thu, Jan 16, 2020 at 2:01 AM jincheng sun < >> sunjincheng...@gmail.com> wrote: >> >>> > >> >>> > I found that there are several dependencies shaded and planned to >> made as vendored artifacts in [1]. I'm not sure why Avro is not shaded >> before. From my point of view, it's a good idea to shade Avro and make it a >> vendored artifact if there are no special reasons blocking us to do that. >> Regarding to how to create a vendored artifact, you can refer to [2] for >> more details. >> >>> > >> >>> > Best, >> >>> > Jincheng >> >>> > >> >>> > [1] https://issues.apache.org/jira/browse/BEAM-5819 >> >>>
Re: Beam's Avro 1.8.x dependency
Looks like there's some strategy to get to the right solution here and that it may likely involve breaking compatibility. One option for myself would be to strip the Beam JAR of AvroCoder and combine with the old AvroCoder from Beam 2.16 -- this would allow me to upgrade Beam but of course is rather hacky. On second thought, was the breaking change from Beam 2.16->2.17 really necessary? If not, could AvroCoder be restored to a 1.9.x "compatible" implementation and kept this way for the Beam 2.1x version lineage? This seems like a somewhat fair ask given the way that I'm suddenly blocked --- however I do realize this is somewhat of a technicality; ie, Beam 2.16-'s compatibility with my usage of Avro 1.9.x was incidental. But, still, if the changes to AvroCoder weren't necessary, restoring back would unblock me and anyone else using Avro 1.9.x (surely I'm not the only one!?) On Thu, Jan 16, 2020 at 12:22 PM Elliotte Rusty Harold wrote: > Avro does not follow semver. They update the major version when the > serialization format changes and the minor version when the API > changes in a backwards incompatible way. See > https://issues.apache.org/jira/browse/AVRO-2687 > > On Thu, Jan 16, 2020 at 12:50 PM Luke Cwik wrote: > > > > Does avro not follow semantic versioning and upgrading to 1.9 should > have been backwards compatible or does our usage reach into the internals > of avro? > > > > On Thu, Jan 16, 2020 at 6:16 AM Ismaël Mejía wrote: > >> > >> I forgot to explain why the most obvious path (just upgrade Avro to > version > >> 1.9.x) is not a valid long term solution. Other systems Beam runs on > top of > >> (e.g. Spark!) also leak Avro into their core so in the moment Spark > moves up > >> to Avro 1.9.x Spark runner users will be in a really fragile position > where > >> things will work until they don't (similar to Aaron's case) so a > stronger reason > >> to getAvro out of Beam core. > >> > >> > >> On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold < > elh...@ibiblio.org> wrote: > >>> > >>> Shading should be a last resort: > >>> > >>> https://jlbp.dev/JLBP-18.html > >>> > >>> It tends to cause more problems than it solves. At best it's a stopgap > >>> measure when you don't have the resources to fix the real problem. In > >>> this case it sounds like the real issue is that AVRO is not stable. > >>> There are at least three other solutions in a case like this: > >>> > >>> 1. Fix Avro at the root. > >>> 2. Fork Avro and then fix it. > >>> 3. Stop depending on Avro. > >>> > >>> None of these are trivial which is why shading gets considered. > >>> However shading doesn't fix the underlying problems, and ultimately > >>> makes a product as unreliable as its least reliable dependency. :-( > >>> > >>> On Thu, Jan 16, 2020 at 2:01 AM jincheng sun > wrote: > >>> > > >>> > I found that there are several dependencies shaded and planned to > made as vendored artifacts in [1]. I'm not sure why Avro is not shaded > before. From my point of view, it's a good idea to shade Avro and make it a > vendored artifact if there are no special reasons blocking us to do that. > Regarding to how to create a vendored artifact, you can refer to [2] for > more details. > >>> > > >>> > Best, > >>> > Jincheng > >>> > > >>> > [1] https://issues.apache.org/jira/browse/BEAM-5819 > >>> > [2] https://github.com/apache/beam/blob/master/vendor/README.md > >>> > > >>> > > >>> > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： > >>> >> > >>> >> I've been upgrading dependencies around gRPC. This Avro-problem is > >>> >> interesting to me. > >>> >> I'll study BEAM-8388 more tomorrow. > >>> >> > >>> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik > wrote: > >>> >> > > >>> >> > +Tomo Suzuki +jincheng sun > >>> >> > There have been a few contributors upgrading the dependencies and > validating things not breaking by running the majority of the post commit > integration tests and also using the linkage checker to show that we aren't > worse off with respect to our dependency tree. Reaching out to them to help > your is your best bet of getting these upgrades through. > >>> >> > > >>> >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon > wrote: > >>> >> >> > >>> >> >> I meant to mention that we must use Avro 1.9.x as we rely on > some schema resolution fixes not present in 1.8.x - so am indeed blocked. > >>> >> >> > >>> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon > wrote: > >>> >> >>> > >>> >> >>> It looks like Avro version dependency from Beam has come up in > the past [1, 2]. > >>> >> >>> > >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my > usage of Avro 1.9.x. > >>> >> >>> > >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that > 2.17.0 has some dependencies on Avro classes only available in 1.8.x. > >>> >> >>> > >>> >> >>> Wondering if anyone else is similar blocked and what it would > take to prioritize Beam upgrading to 1.9.x or better using a shaded version > so that clients can use their own
Re: Beam's Avro 1.8.x dependency
Avro does not follow semver. They update the major version when the serialization format changes and the minor version when the API changes in a backwards incompatible way. See https://issues.apache.org/jira/browse/AVRO-2687 On Thu, Jan 16, 2020 at 12:50 PM Luke Cwik wrote: > > Does avro not follow semantic versioning and upgrading to 1.9 should have > been backwards compatible or does our usage reach into the internals of avro? > > On Thu, Jan 16, 2020 at 6:16 AM Ismaël Mejía wrote: >> >> I forgot to explain why the most obvious path (just upgrade Avro to version >> 1.9.x) is not a valid long term solution. Other systems Beam runs on top of >> (e.g. Spark!) also leak Avro into their core so in the moment Spark moves up >> to Avro 1.9.x Spark runner users will be in a really fragile position where >> things will work until they don't (similar to Aaron's case) so a stronger >> reason >> to getAvro out of Beam core. >> >> >> On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold >> wrote: >>> >>> Shading should be a last resort: >>> >>> https://jlbp.dev/JLBP-18.html >>> >>> It tends to cause more problems than it solves. At best it's a stopgap >>> measure when you don't have the resources to fix the real problem. In >>> this case it sounds like the real issue is that AVRO is not stable. >>> There are at least three other solutions in a case like this: >>> >>> 1. Fix Avro at the root. >>> 2. Fork Avro and then fix it. >>> 3. Stop depending on Avro. >>> >>> None of these are trivial which is why shading gets considered. >>> However shading doesn't fix the underlying problems, and ultimately >>> makes a product as unreliable as its least reliable dependency. :-( >>> >>> On Thu, Jan 16, 2020 at 2:01 AM jincheng sun >>> wrote: >>> > >>> > I found that there are several dependencies shaded and planned to made as >>> > vendored artifacts in [1]. I'm not sure why Avro is not shaded before. >>> > From my point of view, it's a good idea to shade Avro and make it a >>> > vendored artifact if there are no special reasons blocking us to do that. >>> > Regarding to how to create a vendored artifact, you can refer to [2] for >>> > more details. >>> > >>> > Best, >>> > Jincheng >>> > >>> > [1] https://issues.apache.org/jira/browse/BEAM-5819 >>> > [2] https://github.com/apache/beam/blob/master/vendor/README.md >>> > >>> > >>> > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： >>> >> >>> >> I've been upgrading dependencies around gRPC. This Avro-problem is >>> >> interesting to me. >>> >> I'll study BEAM-8388 more tomorrow. >>> >> >>> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: >>> >> > >>> >> > +Tomo Suzuki +jincheng sun >>> >> > There have been a few contributors upgrading the dependencies and >>> >> > validating things not breaking by running the majority of the post >>> >> > commit integration tests and also using the linkage checker to show >>> >> > that we aren't worse off with respect to our dependency tree. Reaching >>> >> > out to them to help your is your best bet of getting these upgrades >>> >> > through. >>> >> > >>> >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: >>> >> >> >>> >> >> I meant to mention that we must use Avro 1.9.x as we rely on some >>> >> >> schema resolution fixes not present in 1.8.x - so am indeed blocked. >>> >> >> >>> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: >>> >> >>> >>> >> >>> It looks like Avro version dependency from Beam has come up in the >>> >> >>> past [1, 2]. >>> >> >>> >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my >>> >> >>> usage of Avro 1.9.x. >>> >> >>> >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 >>> >> >>> has some dependencies on Avro classes only available in 1.8.x. >>> >> >>> >>> >> >>> Wondering if anyone else is similar blocked and what it would take >>> >> >>> to prioritize Beam upgrading to 1.9.x or better using a shaded >>> >> >>> version so that clients can use their own Avro version for their own >>> >> >>> coding purposes. (Eg, I parse Avro messages from a KafkaIO source >>> >> >>> and need 1.9.x for this but am perfectly happy if Beam's Avro coding >>> >> >>> facilities used a shaded other version.) >>> >> >>> >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling >>> >> >>> community for discussion. >>> >> >>> >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >>> >> >>> [2] https://github.com/apache/beam/pull/9779 >>> >> >>> >>> >> >>> >> >>> >> -- >>> >> Regards, >>> >> Tomo >>> >>> >>> >>> -- >>> Elliotte Rusty Harold >>> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
I forgot to explain why the most obvious path (just upgrade Avro to version 1.9.x) is not a valid long term solution. Other systems Beam runs on top of (e.g. Spark!) also leak Avro into their core so in the moment Spark moves up to Avro 1.9.x Spark runner users will be in a really fragile position where things will work until they don't (similar to Aaron's case) so a stronger reason to getAvro out of Beam core. On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold wrote: > Shading should be a last resort: > > https://jlbp.dev/JLBP-18.html > > It tends to cause more problems than it solves. At best it's a stopgap > measure when you don't have the resources to fix the real problem. In > this case it sounds like the real issue is that AVRO is not stable. > There are at least three other solutions in a case like this: > > 1. Fix Avro at the root. > 2. Fork Avro and then fix it. > 3. Stop depending on Avro. > > None of these are trivial which is why shading gets considered. > However shading doesn't fix the underlying problems, and ultimately > makes a product as unreliable as its least reliable dependency. :-( > > On Thu, Jan 16, 2020 at 2:01 AM jincheng sun > wrote: > > > > I found that there are several dependencies shaded and planned to made > as vendored artifacts in [1]. I'm not sure why Avro is not shaded before. > From my point of view, it's a good idea to shade Avro and make it a > vendored artifact if there are no special reasons blocking us to do that. > Regarding to how to create a vendored artifact, you can refer to [2] for > more details. > > > > Best, > > Jincheng > > > > [1] https://issues.apache.org/jira/browse/BEAM-5819 > > [2] https://github.com/apache/beam/blob/master/vendor/README.md > > > > > > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： > >> > >> I've been upgrading dependencies around gRPC. This Avro-problem is > >> interesting to me. > >> I'll study BEAM-8388 more tomorrow. > >> > >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: > >> > > >> > +Tomo Suzuki +jincheng sun > >> > There have been a few contributors upgrading the dependencies and > validating things not breaking by running the majority of the post commit > integration tests and also using the linkage checker to show that we aren't > worse off with respect to our dependency tree. Reaching out to them to help > your is your best bet of getting these upgrades through. > >> > > >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon > wrote: > >> >> > >> >> I meant to mention that we must use Avro 1.9.x as we rely on some > schema resolution fixes not present in 1.8.x - so am indeed blocked. > >> >> > >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon > wrote: > >> >>> > >> >>> It looks like Avro version dependency from Beam has come up in the > past [1, 2]. > >> >>> > >> >>> I'm currently on Beam 2.16.0, which has been compatible with my > usage of Avro 1.9.x. > >> >>> > >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 > has some dependencies on Avro classes only available in 1.8.x. > >> >>> > >> >>> Wondering if anyone else is similar blocked and what it would take > to prioritize Beam upgrading to 1.9.x or better using a shaded version so > that clients can use their own Avro version for their own coding purposes. > (Eg, I parse Avro messages from a KafkaIO source and need 1.9.x for this > but am perfectly happy if Beam's Avro coding facilities used a shaded other > version.) > >> >>> > >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling > community for discussion. > >> >>> > >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 > >> >>> [2] https://github.com/apache/beam/pull/9779 > >> >>> > >> > >> > >> -- > >> Regards, > >> Tomo > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org >
Re: Beam's Avro 1.8.x dependency
Shading should be a last resort: https://jlbp.dev/JLBP-18.html It tends to cause more problems than it solves. At best it's a stopgap measure when you don't have the resources to fix the real problem. In this case it sounds like the real issue is that AVRO is not stable. There are at least three other solutions in a case like this: 1. Fix Avro at the root. 2. Fork Avro and then fix it. 3. Stop depending on Avro. None of these are trivial which is why shading gets considered. However shading doesn't fix the underlying problems, and ultimately makes a product as unreliable as its least reliable dependency. :-( On Thu, Jan 16, 2020 at 2:01 AM jincheng sun wrote: > > I found that there are several dependencies shaded and planned to made as > vendored artifacts in [1]. I'm not sure why Avro is not shaded before. From > my point of view, it's a good idea to shade Avro and make it a vendored > artifact if there are no special reasons blocking us to do that. Regarding to > how to create a vendored artifact, you can refer to [2] for more details. > > Best, > Jincheng > > [1] https://issues.apache.org/jira/browse/BEAM-5819 > [2] https://github.com/apache/beam/blob/master/vendor/README.md > > > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： >> >> I've been upgrading dependencies around gRPC. This Avro-problem is >> interesting to me. >> I'll study BEAM-8388 more tomorrow. >> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: >> > >> > +Tomo Suzuki +jincheng sun >> > There have been a few contributors upgrading the dependencies and >> > validating things not breaking by running the majority of the post commit >> > integration tests and also using the linkage checker to show that we >> > aren't worse off with respect to our dependency tree. Reaching out to them >> > to help your is your best bet of getting these upgrades through. >> > >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: >> >> >> >> I meant to mention that we must use Avro 1.9.x as we rely on some schema >> >> resolution fixes not present in 1.8.x - so am indeed blocked. >> >> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: >> >>> >> >>> It looks like Avro version dependency from Beam has come up in the past >> >>> [1, 2]. >> >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my usage of >> >>> Avro 1.9.x. >> >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has >> >>> some dependencies on Avro classes only available in 1.8.x. >> >>> >> >>> Wondering if anyone else is similar blocked and what it would take to >> >>> prioritize Beam upgrading to 1.9.x or better using a shaded version so >> >>> that clients can use their own Avro version for their own coding >> >>> purposes. (Eg, I parse Avro messages from a KafkaIO source and need >> >>> 1.9.x for this but am perfectly happy if Beam's Avro coding facilities >> >>> used a shaded other version.) >> >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling >> >>> community for discussion. >> >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >> >>> [2] https://github.com/apache/beam/pull/9779 >> >>> >> >> >> -- >> Regards, >> Tomo -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
Adding to Ismaël, I find moving Avro out of the core, and keeping compatibility as a non-exclusive options. Of course, it would require more effort from our side. On Thu, Jan 16, 2020 at 12:29 PM Ismaël Mejía wrote: > For the interested there was also some extra context in the discussion at: > https://github.com/apache/beam/pull/9779 > > Gleb mentioned the key two points: > > 1. The fact that Avro is exposed in the User API in beam-sdks-java-core > was a >mistake and makes fixing this issue backwards incompatible. > > 2. Shading is not an option because Avro compiler would generate specific >records that won't match the non-vendored version so we will break user >records compatibility (for example for users with a schema registry). > > So save if I am missing something and someone can give an alternative, we > are > in a situation where the only solution to the issue is to do (1), move > Avro out > of core as an extension but then the question is would we sacrifice > breaking > backwards compatibility for this issue. I am in the 'we should do it' camp. > What do others think? > > > On Thu, Jan 16, 2020 at 10:17 AM Gleb Kanterov wrote: > >> There are significant changes between Avro 1.8 and Avro 1.9. I'm not sure >> it's possible for beam-sdks-java-core to support both at the same time. The >> fact that AvroIO is a part of the beam-sdks-java-core doesn't make it >> simpler. However, I can see how we can build two binary artifacts with the >> same user-facing API each supporting own version of Avro. >> >> Shading or vendoring would be a breaking change because public signatures >> of AvroIO (and few other IOs, for instance, BigQueryIO) refer to classes >> from Avro, for instance, GenericRecord. Furthermore, a lot of Beam users >> use Avro compiler to generate Java code for SpecificRecord, which would >> refer to non-vendored version. >> >
Re: Beam's Avro 1.8.x dependency
For the interested there was also some extra context in the discussion at: https://github.com/apache/beam/pull/9779 Gleb mentioned the key two points: 1. The fact that Avro is exposed in the User API in beam-sdks-java-core was a mistake and makes fixing this issue backwards incompatible. 2. Shading is not an option because Avro compiler would generate specific records that won't match the non-vendored version so we will break user records compatibility (for example for users with a schema registry). So save if I am missing something and someone can give an alternative, we are in a situation where the only solution to the issue is to do (1), move Avro out of core as an extension but then the question is would we sacrifice breaking backwards compatibility for this issue. I am in the 'we should do it' camp. What do others think? On Thu, Jan 16, 2020 at 10:17 AM Gleb Kanterov wrote: > There are significant changes between Avro 1.8 and Avro 1.9. I'm not sure > it's possible for beam-sdks-java-core to support both at the same time. The > fact that AvroIO is a part of the beam-sdks-java-core doesn't make it > simpler. However, I can see how we can build two binary artifacts with the > same user-facing API each supporting own version of Avro. > > Shading or vendoring would be a breaking change because public signatures > of AvroIO (and few other IOs, for instance, BigQueryIO) refer to classes > from Avro, for instance, GenericRecord. Furthermore, a lot of Beam users > use Avro compiler to generate Java code for SpecificRecord, which would > refer to non-vendored version. >
Re: Beam's Avro 1.8.x dependency
There are significant changes between Avro 1.8 and Avro 1.9. I'm not sure it's possible for beam-sdks-java-core to support both at the same time. The fact that AvroIO is a part of the beam-sdks-java-core doesn't make it simpler. However, I can see how we can build two binary artifacts with the same user-facing API each supporting own version of Avro. Shading or vendoring would be a breaking change because public signatures of AvroIO (and few other IOs, for instance, BigQueryIO) refer to classes from Avro, for instance, GenericRecord. Furthermore, a lot of Beam users use Avro compiler to generate Java code for SpecificRecord, which would refer to non-vendored version.
Re: Beam's Avro 1.8.x dependency
I found that there are several dependencies shaded and planned to made as vendored artifacts in [1]. I'm not sure why Avro is not shaded before. From my point of view, it's a good idea to shade Avro and make it a vendored artifact if there are no special reasons blocking us to do that. Regarding to how to create a vendored artifact, you can refer to [2] for more details. Best, Jincheng [1] https://issues.apache.org/jira/browse/BEAM-5819 [2] https://github.com/apache/beam/blob/master/vendor/README.md Tomo Suzuki 于2020年1月16日周四 下午1:18写道： > I've been upgrading dependencies around gRPC. This Avro-problem is > interesting to me. > I'll study BEAM-8388 more tomorrow. > > On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: > > > > +Tomo Suzuki +jincheng sun > > There have been a few contributors upgrading the dependencies and > validating things not breaking by running the majority of the post commit > integration tests and also using the linkage checker to show that we aren't > worse off with respect to our dependency tree. Reaching out to them to help > your is your best bet of getting these upgrades through. > > > > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: > >> > >> I meant to mention that we must use Avro 1.9.x as we rely on some > schema resolution fixes not present in 1.8.x - so am indeed blocked. > >> > >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: > >>> > >>> It looks like Avro version dependency from Beam has come up in the > past [1, 2]. > >>> > >>> I'm currently on Beam 2.16.0, which has been compatible with my usage > of Avro 1.9.x. > >>> > >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 > has some dependencies on Avro classes only available in 1.8.x. > >>> > >>> Wondering if anyone else is similar blocked and what it would take to > prioritize Beam upgrading to 1.9.x or better using a shaded version so that > clients can use their own Avro version for their own coding purposes. (Eg, > I parse Avro messages from a KafkaIO source and need 1.9.x for this but am > perfectly happy if Beam's Avro coding facilities used a shaded other > version.) > >>> > >>> I've made a comment on BEAM-8388 [1] to this effect. But polling > community for discussion. > >>> > >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 > >>> [2] https://github.com/apache/beam/pull/9779 > >>> > > > -- > Regards, > Tomo >
Re: Beam's Avro 1.8.x dependency
I've been upgrading dependencies around gRPC. This Avro-problem is interesting to me. I'll study BEAM-8388 more tomorrow. On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: > > +Tomo Suzuki +jincheng sun > There have been a few contributors upgrading the dependencies and validating > things not breaking by running the majority of the post commit integration > tests and also using the linkage checker to show that we aren't worse off > with respect to our dependency tree. Reaching out to them to help your is > your best bet of getting these upgrades through. > > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: >> >> I meant to mention that we must use Avro 1.9.x as we rely on some schema >> resolution fixes not present in 1.8.x - so am indeed blocked. >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: >>> >>> It looks like Avro version dependency from Beam has come up in the past [1, >>> 2]. >>> >>> I'm currently on Beam 2.16.0, which has been compatible with my usage of >>> Avro 1.9.x. >>> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has >>> some dependencies on Avro classes only available in 1.8.x. >>> >>> Wondering if anyone else is similar blocked and what it would take to >>> prioritize Beam upgrading to 1.9.x or better using a shaded version so that >>> clients can use their own Avro version for their own coding purposes. (Eg, >>> I parse Avro messages from a KafkaIO source and need 1.9.x for this but am >>> perfectly happy if Beam's Avro coding facilities used a shaded other >>> version.) >>> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling community >>> for discussion. >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >>> [2] https://github.com/apache/beam/pull/9779 >>> -- Regards, Tomo
Re: Beam's Avro 1.8.x dependency
+Tomo Suzuki +jincheng sun There have been a few contributors upgrading the dependencies and validating things not breaking by running the majority of the post commit integration tests and also using the linkage checker to show that we aren't worse off with respect to our dependency tree. Reaching out to them to help your is your best bet of getting these upgrades through. On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: > I meant to mention that we must use Avro 1.9.x as we rely on some schema > resolution fixes not present in 1.8.x - so am indeed blocked. > > On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: > >> It looks like Avro version dependency from Beam has come up in the past >> [1, 2]. >> >> I'm currently on Beam 2.16.0, which has been compatible with my usage of >> Avro 1.9.x. >> >> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has >> some dependencies on Avro classes only available in 1.8.x. >> >> Wondering if anyone else is similar blocked and what it would take to >> prioritize Beam upgrading to 1.9.x or better using a shaded version so that >> clients can use their own Avro version for their own coding purposes. (Eg, >> I parse Avro messages from a KafkaIO source and need 1.9.x for this but am >> perfectly happy if Beam's Avro coding facilities used a shaded other >> version.) >> >> I've made a comment on BEAM-8388 [1] to this effect. But polling >> community for discussion. >> >> [1] https://issues.apache.org/jira/browse/BEAM-8388 >> [2] https://github.com/apache/beam/pull/9779 >> >>
Re: Beam's Avro 1.8.x dependency
I meant to mention that we must use Avro 1.9.x as we rely on some schema resolution fixes not present in 1.8.x - so am indeed blocked. On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: > It looks like Avro version dependency from Beam has come up in the past > [1, 2]. > > I'm currently on Beam 2.16.0, which has been compatible with my usage of > Avro 1.9.x. > > But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has > some dependencies on Avro classes only available in 1.8.x. > > Wondering if anyone else is similar blocked and what it would take to > prioritize Beam upgrading to 1.9.x or better using a shaded version so that > clients can use their own Avro version for their own coding purposes. (Eg, > I parse Avro messages from a KafkaIO source and need 1.9.x for this but am > perfectly happy if Beam's Avro coding facilities used a shaded other > version.) > > I've made a comment on BEAM-8388 [1] to this effect. But polling community > for discussion. > > [1] https://issues.apache.org/jira/browse/BEAM-8388 > [2] https://github.com/apache/beam/pull/9779 > >
