Hello Ismaël, Thank you for the clarification and work on these issues!
I submitted a similar issue for commons-compress [1]; though it appears they are still on 1.4.8-7 [2] hopefully they'll also be on 1.4.9-1 before the next release. michael [1] https://issues.apache.org/jira/browse/COMPRESS-564 <https://issues.apache.org/jira/browse/COMPRESS-564> [2] https://github.com/apache/commons-compress/pull/174 <https://github.com/apache/commons-compress/pull/174> > On Mar 10, 2021, at 1:25 PM, Ismaël Mejía <[email protected]> wrote: > > Thanks for bringing this subject Michael, we have vastly improved our > communication with downstream projects (Spark/Parquet/Hive) and most > of the issues you and other users have had will be available soon in > upcoming versions of those projects. > > Spark finally merged the upgrade to Avro 1.10.1 recently [1], the > Parquet upgrade will be there too. Dongjoon (one Spark committer) has > been contributing to the zstd upgrades in the three projects > (Avro/Parquet/Spark) so we should be good to go. Actually the 1.10.2 > release is motivated in part because of the needs of Spark [2] I > expect to open the PR with the Avro upgrade once published so it will > be included in Spark 3.2. > > Regards, > Ismaël > > [1] https://github.com/apache/spark/pull/31232 > [2] https://github.com/apache/avro/pull/1129 > > On Wed, Mar 10, 2021 at 7:28 PM Michael Heuer <[email protected]> wrote: >> >> Hello, >> >> The zstd-jni dependency version has been moving quickly lately, and there >> are API changes (although hopefully all compatible). Any worry about >> version conflicts between Avro, Parquet, Spark, commons-compress, and other >> projects that all have a dependency on zstd-jni? >> >> I would test this RC with our stack, but we're still stuck on Avro 1.8.2 due >> to other version conflicts between Avro and Parquet in Spark (which I >> believe have been resolved in the master branch towards a future Spark 3.2 >> release). >> >> Sorry I can't be more help other than raising a possible concern. >> >> michael >> >> >>> On Mar 10, 2021, at 12:17 PM, Ryan Skraba <[email protected]> wrote: >>> >>> Hello! >>> >>> I finished some validation and I'm pleased to add my own vote on the >>> release candidate! >>> >>> I checked the Avro 1.10.2-rc2 artifacts by: >>> >>> * Running a large set of internal Java code and unit tests (already on >>> 1.10) with these candidates. >>> * Building parquet-mr with the candidates. >>> * We have some internal python code and unit tests that I ran on versions >>> python3.5 - python3.9. >>> * Checked that all of the artifacts were signed and verified my own >>> signature (FWIW) >>> * Checked that the artifacts in the staging nexus were signed by my own >>> signature (FWIW) >>> >>> It looks good for me! >>> >>> [X] +1 Release this as Apache Avro 1.10.2 >>> [ ] +0 >>> [ ] -1 Do not release this because... >>> >>> All my best, Ryan >>> >>> >>> On Tue, Mar 9, 2021 at 6:55 PM Ryan Skraba <[email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> I'd like to propose the following RC2 to be released as the official Apache >>>> Avro 1.10.2 release. >>>> >>>> There have been two commits since RC1, addressing the following issues: >>>> - [AVRO-3072] ZSTD Performance improvements >>>> - [AVRO-3073] Potential vulnerability with velocity >>>> >>>> The commit id is 8111cdc35430ff68dcb644306362859de40999d9 >>>> * This corresponds to the tag: release-1.10.2-rc2 >>>> * https://github.com/apache/avro/releases/tag/release-1.10.2-rc2 >>>> >>>> The release tarball, signature, and checksums are here (revision 46543): >>>> * https://dist.apache.org/repos/dist/dev/avro/avro-1.10.2-rc2/ >>>> >>>> You can find the KEYS file here: >>>> * https://dist.apache.org/repos/dist/dev/avro/KEYS >>>> >>>> Binary artifacts for Java are staged in Nexus here: >>>> * >>>> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.10.2/ >>>> >>>> This release includes ~30 Jira issues: >>>> * >>>> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.2 >>>> >>>> Some interesting highlights: >>>> >>>> Avro specification >>>> - [AVRO-3028] Clarify that records encode values even if they equal their >>>> default >>>> >>>> C# >>>> - [AVRO-3005][AVRO-2983] BinaryDecoder fails to read large strings >>>> >>>> C++ >>>> - [AVRO-3031] avrocppgen does not generate correct C++ code when the >>>> schema contains >>>> C++ reserved words >>>> >>>> Java >>>> - [AVRO-2471] Java code generation doesn't add conversion for >>>> timestamp-micros >>>> - [AVRO-2860] More Closely Adhere to ASF Parent POM >>>> - [AVRO-2944] DataFileReader has incorrect logic reading magic header >>>> - [AVRO-3024] Bump Jackson to 2.12.1 >>>> - [AVRO-3060] Support ZSTD level and BufferPool options >>>> - [AVRO-3049] BinaryDecoder lacks checks on bytes array length >>>> - [AVRO-3072] ZSTD Performance improvements >>>> >>>> Python >>>> - [AVRO-3006] Update PyPI documentation to deprecate avro-python3 >>>> *** The avro package supports Python 3, and avro-python3 will be removed >>>> in >>>> the next major release *** >>>> >>>> Ruby >>>> - [AVRO-2984] Unnecessary memory allocations during serialization >>>> - [AVRO-2998] Records with symbol keys fail validation >>>> - [AVRO-2999] Optimize Ruby union serialization >>>> - [AVRO-3000] Avoid unnecessary schema compatibility checks >>>> - [AVRO-3023] Validate with Ruby 3 >>>> >>>> * Upgrade dependencies to latest versions, including CVE fixes. >>>> * Multiple fixes, better documentation and more... >>>> >>>> Avro 1.10 is still using Travis, but the status isn't necessarily >>>> reflected on the branch in github: >>>> * https://travis-ci.com/github/apache/avro/builds/219514042 >>>> >>>> Please download, verify, and test. This vote will remain open for at least >>>> 72 hours. Given sufficient votes, I would like to close after the weekend >>>> on >>>> 18h00 UTC Friday, March 12th, 2021 >>>> >>>> [ ] +1 Release this as Apache Avro 1.10.2 >>>> [ ] +0 >>>> [ ] -1 Do not release this because... >>>> >>>> Best regards, >>>> Ryan Skraba >>>> >>
