Alright, it looks like something was broken with my setup. I think a straightforward upgrade is actually possible, so I'm going to continue on that
On Wed, Nov 15, 2023 at 10:17 AM John Casey <theotherj...@google.com> wrote: > So, thats the thing. I've upgraded to 1.11.3, but the plugin still seems > to be generating Avro code that isn't compatible with 1.11.3. > > Looking at the PR https://github.com/apache/beam/pull/29390, it looks > like it generates avro code based on 1.9.2, which ends up being > incompatible. > > I don't think we can continue to support every avro version with our > current setup. > > John > > On Tue, Nov 14, 2023 at 4:20 PM Alexey Romanenko <aromanenko....@gmail.com> > wrote: > >> Thanks! Please, let me know if you need any help on this. >> >> — >> Alexey >> >> On 14 Nov 2023, at 17:52, John Casey <theotherj...@google.com> wrote: >> >> The vulnerability said to upgrade to 1.11.3, so I think that would be my >> starting point. >> >> >> On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko < >> aromanenko....@gmail.com> wrote: >> >>> >>> >>> On 10 Nov 2023, at 19:23, John Casey <theotherj...@google.com> wrote: >>> >>> I guess I'm a bit confused as to why specifically generateTestAvroJava >>> seems to use the wrong version. I see our version specific generated code, >>> but this action appears to be inherited from the plugin, and is configured >>> with whichever avro version is provided. Given that I tried to just change >>> to 1.11.3, I'm confused as to why its generating invalid java files for the >>> provided avro version. >>> >>> Unlike the classes generated out of the JavaExec you referenced, this >>> appears to only generate one version of the files. >>> >>> >>> It was supposed to generate files with a specific Avro version every >>> time to run the same tests again this specific Avro version. >>> >>> It may be that we don't need this action, but it still seems to run, as >>> we depend on it in the applyAvroNature() action. >>> >>> >>> I started to think if we really still need this action. >>> >>> We could remove this entirely. The java exec only generates versions for >>> pre-configured test versions anyways >>> >>> >>> Right. The point is in how many places in Beam we need to generate these >>> files and which version(s) of Avro to use? >>> >>> — >>> Alexey >>> >>> >>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko < >>> aromanenko....@gmail.com> wrote: >>> >>>> Hi John, >>>> >>>> This old Avro version in Beam is a very long story. Briefly, since >>>> initially it was toughly integrated into Java SDK “core” module then it was >>>> not possible to upgrade an Avro version without breaking changes for users >>>> (because of some Avro incompatible changes, as you have noticed before). >>>> So, we decided to extract Avro-related classes from Beam “core” to a >>>> dedicated Avro extension [2] that supports and actually is tested with >>>> different Avro versions. More details on this work are here [1] >>>> >>>> Regarding auto-generated classes. Initially, we used a Gradle plugin >>>> for that but it’s limited with only one Avro version per instance of this >>>> plugin, so it was not possible to generate these classes with different >>>> Avro versions. So, we do this with a special Gradle task (“JavaExec") that >>>> executes “org.apache.avro.tool.Main” and generate Avro classes per every >>>> tested Avro version [3]. >>>> >>>> We still keep an old Avro version 1.8.2. as a default dependency >>>> version but it will be overwritten if users have a more recent one as a >>>> project dependency in their classpath. >>>> >>>> I think we need to completely remove Avro Gradle plugin (use “JavaExec” >>>> task to generate Avro classes with a provided Avro version instead) and >>>> update the default Avro version to the more recent one since now it’s not >>>> part of Java “core”. >>>> >>>> Any thoughts? >>>> >>>> — >>>> Alexey >>>> >>>> >>>> [1] https://github.com/apache/beam/issues/24292 >>>> [2] >>>> https://github.com/apache/beam/tree/master/sdks/java/extensions/avro >>>> [3] >>>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 >>>> >>>> >>>> >>>> On 10 Nov 2023, at 18:05, John Casey via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>> Hi All, >>>> >>>> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying >>>> to upgrade to avro 1.11.3. >>>> >>>> Unfortunately, it seems that our auto-generated Avro test classes >>>> aren't being generated properly with this new version. I've updated our >>>> avro generation plugin as well, but for whatever reason, it seems that the >>>> generated AvroTest file is being generated with references to classes that >>>> did exist in 1.8.2, but no longer exist in 1.11.3. >>>> >>>> It seems like our autogeneration is being run with the wrong avro >>>> version, but I can't seem to find where that would be configured. >>>> >>>> Here is the PR with my changes so far: >>>> https://github.com/apache/beam/pull/29390 >>>> >>>> Is anyone familiar with what might be misconfigured here? >>>> >>>> John >>>> >>>> >>>> >>> >>
