Hello Ryan, If you don't mind me asking, what were the changes you found necessary for Parquet to work with Avro 1.10?
The only reason I lurk on this mailing list is in the vain hope that Parquet and Avro and Spark will work together nicely at some point. michael > On Jun 12, 2020, at 3:44 AM, Ryan Skraba <[email protected]> wrote: > > Hello! > > I checked the signature and SHA512 sums of the artifacts. > > Internally we have a large library of internal code that uses Avro > 1.9, and it all recompiled and tests succeeded without modification. > > We have some other code that definitely breaks with 1.10 because of > AVRO-2278 (getting an unknown field throws an exception). In > practice, it's not going to impact us in the short term, since it's > all run on a cluster where the avro version is provided (and usually > 1.8.x). Our code will need to be rewritten to handle these exceptions > gracefully for Avro 1.10+ > > Parquet needed some minor massaging because of the change to the > avro-maven-plugin (now generates private field members). This should > definitely be in the release notes: > > AVRO-2581: Maven plugin generates specific records with private fields. > > The arrow Avro adapter builds and tests with the 1.10.0 RC artifacts. > > I did some minor tests with a toy python project (both py and py3). > > +1 for me (mostly Java, some python). > > If there is a blocker or another release candidate, AVRO-2849 (spec > clarification) would be nice to include. > > All my best and thanks for the work, Ismael! > > Ryan > > > On Thu, Jun 11, 2020 at 2:25 PM Daniel Kulp <[email protected]> wrote: >> >> +1 >> >> Been playing around with it a bit and it looks OK. >> >> Dan >> >> >>> On Jun 11, 2020, at 4:53 AM, Ismaël Mejía <[email protected]> wrote: >>> >>> Kind reminder two days have passed, please check and vote! >>> >>> On Thu, Jun 11, 2020 at 2:29 AM Andy Le <[email protected]> wrote: >>>> >>>> Hey Ryan, >>>> >>>> Thank you for your clear explanation. >>>> >>>> +1 for me :D >>>> >>>> Thank you. >>>> >>>> >>>> On 2020/06/10 14:39:10, Ryan Skraba <[email protected]> wrote: >>>>> Hey Anh, a release requires 3 PMC votes, but "unofficial" opinions are >>>>> welcome and taken into account. My opinion is unofficial too! >>>>> >>>>> Just reply with a +1 if you think the artifacts are good to go or -1 >>>>> if you've found a bug, regression, security issue, or something that >>>>> should block these artifacts from being the 1.10.0 release. >>>>> >>>>> For example, at my company, we have a lot of internal code and >>>>> libraries that work with Avro 1.9.2, especially the generic Java APIs, >>>>> so I'm going to compile with 1.10.0 and see if all the unit tests >>>>> pass. >>>>> >>>>> If you're familiar with building any open source projects that use >>>>> Avro, try bumping there and reporting back. You can go through the >>>>> open JIRA and open PR and make recommendations if we missed something >>>>> important. >>>>> >>>>> For example, I've gone through a lot of the JIRA and it looks like >>>>> AVRO-2799 (a protobuf bug) is blocking a user. I'm looking to see if >>>>> this is a regression or if it's an easy fix before I vote. >>>>> >>>>> I've verified the signatures and SHA512 of the artifacts, so that's good! >>>>> >>>>> All my best, Ryan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jun 10, 2020 at 8:53 AM Anh Le <[email protected]> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> How can I make a vote on this? >>>>>> >>>>>> Best Regards. >>>>>> >>>>>> On Tue, 9 Jun 2020, 22:28 Ismaël Mejía, <[email protected]> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> I'd like to propose the following RC1 to be released as the official >>>>>>> Apache >>>>>>> Avro 1.10.0 release. >>>>>>> >>>>>>> The commit id is 1301a045396cf0e6ca7f12c9e4053b4f81237887 >>>>>>> >>>>>>> * This corresponds to the tag: release-1.10.0-rc1 >>>>>>> * https://github.com/apache/avro/releases/tag/release-1.10.0-rc1 >>>>>>> >>>>>>> The release tarball, signature, and checksums are here (revision 37932): >>>>>>> * https://dist.apache.org/repos/dist/dev/avro/avro-1.10.0-rc1/ >>>>>>> >>>>>>> You can find the KEYS file here: >>>>>>> * https://dist.apache.org/repos/dist/dev/avro/KEYS >>>>>>> >>>>>>> Binary artifacts for Java are staged in Nexus here: >>>>>>> * >>>>>>> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.10.0/ >>>>>>> >>>>>>> This release includes 187 Jira issues (Thanks Ryan Skraba for the >>>>>>> highlights) >>>>>>> >>>>>>> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.0 >>>>>>> >>>>>>> * Some interesting highlights (Thanks Ryan Skraba for the list): >>>>>>> >>>>>>> C# >>>>>>> - [AVRO-2389] Add Avro serialization for POCO (Reflection) >>>>>>> >>>>>>> Java >>>>>>> - [AVRO-2723] Automatically find defaults on POJO when using reflection. >>>>>>> - [AVRO-2278] Now throws an exception when getting a non-existent >>>>>>> field from a record >>>>>>> - [AVRO-2335] Remove Joda Time library. >>>>>>> - [AVRO-2438] Better support for URI and URL types. >>>>>>> >>>>>>> Perl >>>>>>> - [AVRO-1461] Distribute as a CPAN module. >>>>>>> >>>>>>> PHP >>>>>>> - [AVRO-2527] Update to PHP 7.x >>>>>>> >>>>>>> Python >>>>>>> - [AVRO-2656] avro-python package is now the preferred python3 library >>>>>>> and avro-python3 is prepared to be deprecated >>>>>>> - [AVRO-2387] type checking added to python >>>>>>> >>>>>>> Ruby >>>>>>> - [AVRO-1740] Support fingerprinting. >>>>>>> - [AVRO-2535] Support enum defaults. >>>>>>> - [AVRO-2545] Support aliases. >>>>>>> >>>>>>> * Upgrade dependencies to latest versions, including CVE fixes. >>>>>>> * Multiple fixes and more... >>>>>>> >>>>>>> Please download, verify, and test. This vote will remain open for at >>>>>>> least >>>>>>> 72 hours. Given sufficient votes, I would like to close after noon UTC >>>>>>> Friday, June 12th, 2020 >>>>>>> >>>>>>> [ ] +1 Release this as Apache Avro 1.10.0 >>>>>>> [ ] +0 >>>>>>> [ ] -1 Do not release this because... >>>>>>> >>>>>>> Best regards, >>>>>>> Ismaël Mejía >>>>>>> >>>>> >> >> -- >> Daniel Kulp >> [email protected] <mailto:[email protected]> - http://dankulp.com/blog >> <http://dankulp.com/blog> >> Talend Community Coder - http://talend.com <http://coders.talend.com/>
