As I mentioned before, we always can generate version-dependent Avro classes by running “org.apache.avro.tool.Main" directly with “JavaExec” Gradle task.
Please, see this implementation in Avro extension: https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 In this case, we don’t depend on any 3rd party plugin, that can be run only with one Avro version by the whole project iirc. — Alexey > On 15 Nov 2023, at 17:14, John Casey <theotherj...@google.com> wrote: > > Alright, it looks like something was broken with my setup. I think a > straightforward upgrade is actually possible, so I'm going to continue on that > > On Wed, Nov 15, 2023 at 10:17 AM John Casey <theotherj...@google.com > <mailto:theotherj...@google.com>> wrote: >> So, thats the thing. I've upgraded to 1.11.3, but the plugin still seems to >> be generating Avro code that isn't compatible with 1.11.3. >> >> Looking at the PR https://github.com/apache/beam/pull/29390, it looks like >> it generates avro code based on 1.9.2, which ends up being incompatible. >> >> I don't think we can continue to support every avro version with our current >> setup. >> >> John >> >> On Tue, Nov 14, 2023 at 4:20 PM Alexey Romanenko <aromanenko....@gmail.com >> <mailto:aromanenko....@gmail.com>> wrote: >>> Thanks! Please, let me know if you need any help on this. >>> >>> — >>> Alexey >>> >>>> On 14 Nov 2023, at 17:52, John Casey <theotherj...@google.com >>>> <mailto:theotherj...@google.com>> wrote: >>>> >>>> The vulnerability said to upgrade to 1.11.3, so I think that would be my >>>> starting point. >>>> >>>> >>>> On Mon, Nov 13, 2023 at 12:23 PM Alexey Romanenko >>>> <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: >>>>> >>>>> >>>>>> On 10 Nov 2023, at 19:23, John Casey <theotherj...@google.com >>>>>> <mailto:theotherj...@google.com>> wrote: >>>>>> >>>>>> I guess I'm a bit confused as to why specifically generateTestAvroJava >>>>>> seems to use the wrong version. I see our version specific generated >>>>>> code, but this action appears to be inherited from the plugin, and is >>>>>> configured with whichever avro version is provided. Given that I tried >>>>>> to just change to 1.11.3, I'm confused as to why its generating invalid >>>>>> java files for the provided avro version. >>>>>> >>>>>> Unlike the classes generated out of the JavaExec you referenced, this >>>>>> appears to only generate one version of the files. >>>>> >>>>> It was supposed to generate files with a specific Avro version every time >>>>> to run the same tests again this specific Avro version. >>>>> >>>>>> It may be that we don't need this action, but it still seems to run, as >>>>>> we depend on it in the applyAvroNature() action. >>>>> >>>>> I started to think if we really still need this action. >>>>> >>>>>> We could remove this entirely. The java exec only generates versions for >>>>>> pre-configured test versions anyways >>>>> >>>>> Right. The point is in how many places in Beam we need to generate these >>>>> files and which version(s) of Avro to use? >>>>> >>>>> — >>>>> Alexey >>>>> >>>>>> >>>>>> On Fri, Nov 10, 2023 at 12:53 PM Alexey Romanenko >>>>>> <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: >>>>>>> Hi John, >>>>>>> >>>>>>> This old Avro version in Beam is a very long story. Briefly, since >>>>>>> initially it was toughly integrated into Java SDK “core” module then it >>>>>>> was not possible to upgrade an Avro version without breaking changes >>>>>>> for users (because of some Avro incompatible changes, as you have >>>>>>> noticed before). So, we decided to extract Avro-related classes from >>>>>>> Beam “core” to a dedicated Avro extension [2] that supports and >>>>>>> actually is tested with different Avro versions. More details on this >>>>>>> work are here [1] >>>>>>> >>>>>>> Regarding auto-generated classes. Initially, we used a Gradle plugin >>>>>>> for that but it’s limited with only one Avro version per instance of >>>>>>> this plugin, so it was not possible to generate these classes with >>>>>>> different Avro versions. So, we do this with a special Gradle task >>>>>>> (“JavaExec") that executes “org.apache.avro.tool.Main” and generate >>>>>>> Avro classes per every tested Avro version [3]. >>>>>>> >>>>>>> We still keep an old Avro version 1.8.2. as a default dependency >>>>>>> version but it will be overwritten if users have a more recent one as a >>>>>>> project dependency in their classpath. >>>>>>> >>>>>>> I think we need to completely remove Avro Gradle plugin (use “JavaExec” >>>>>>> task to generate Avro classes with a provided Avro version instead) and >>>>>>> update the default Avro version to the more recent one since now it’s >>>>>>> not part of Java “core”. >>>>>>> >>>>>>> Any thoughts? >>>>>>> >>>>>>> — >>>>>>> Alexey >>>>>>> >>>>>>> >>>>>>> [1] https://github.com/apache/beam/issues/24292 >>>>>>> [2] https://github.com/apache/beam/tree/master/sdks/java/extensions/avro >>>>>>> [3] >>>>>>> https://github.com/apache/beam/blob/c713425e1ac2cdc3ec2ec264c9bf61f7356856bd/sdks/java/extensions/avro/build.gradle#L135 >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On 10 Nov 2023, at 18:05, John Casey via dev <dev@beam.apache.org >>>>>>>> <mailto:dev@beam.apache.org>> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying >>>>>>>> to upgrade to avro 1.11.3. >>>>>>>> >>>>>>>> Unfortunately, it seems that our auto-generated Avro test classes >>>>>>>> aren't being generated properly with this new version. I've updated >>>>>>>> our avro generation plugin as well, but for whatever reason, it seems >>>>>>>> that the generated AvroTest file is being generated with references to >>>>>>>> classes that did exist in 1.8.2, but no longer exist in 1.11.3. >>>>>>>> >>>>>>>> It seems like our autogeneration is being run with the wrong avro >>>>>>>> version, but I can't seem to find where that would be configured. >>>>>>>> >>>>>>>> Here is the PR with my changes so far: >>>>>>>> https://github.com/apache/beam/pull/29390 >>>>>>>> >>>>>>>> Is anyone familiar with what might be misconfigured here? >>>>>>>> >>>>>>>> John >>>>>>> >>>>> >>>
