The only change necessary to compile Parquet with the 1.10.0 RC1 artifacts was to add `<fieldVisibility>public</fieldVisibility>` to the parquet-avro pom.xml so that field members are generated with public access (the default is now private). Or we could rewrite the test classes to use getters!
(Just for info, if you want to reproduce my build, I also changed the repositories and pluginRepositories to temporarily include the https://repository.apache.org/content/groups/staging for these release candidates. Something in my settings makes protoc-jar-maven-plugin unhappy with this setup, and the workaround is to build to a brand new local m2: `-Dmaven.repo.local=/tmp/m2` ) I'm interested in your Parquet/Avro/Spark experience! I've gotten them to work together using Apache Beam on Spark and extensive shading :/ Ismael has done some excellent (and tricky!) work trying to get the communities to move forward together! All my best, Ryan On Fri, Jun 12, 2020 at 7:23 PM Michael Heuer <[email protected]> wrote: > > Hello Ryan, > > If you don't mind me asking, what were the changes you found necessary for > Parquet to work with Avro 1.10? > > The only reason I lurk on this mailing list is in the vain hope that Parquet > and Avro and Spark will work together nicely at some point. > > michael > > > > On Jun 12, 2020, at 3:44 AM, Ryan Skraba <[email protected]> wrote: > > > > Hello! > > > > I checked the signature and SHA512 sums of the artifacts. > > > > Internally we have a large library of internal code that uses Avro > > 1.9, and it all recompiled and tests succeeded without modification. > > > > We have some other code that definitely breaks with 1.10 because of > > AVRO-2278 (getting an unknown field throws an exception). In > > practice, it's not going to impact us in the short term, since it's > > all run on a cluster where the avro version is provided (and usually > > 1.8.x). Our code will need to be rewritten to handle these exceptions > > gracefully for Avro 1.10+ > > > > Parquet needed some minor massaging because of the change to the > > avro-maven-plugin (now generates private field members). This should > > definitely be in the release notes: > > > > AVRO-2581: Maven plugin generates specific records with private fields. > > > > The arrow Avro adapter builds and tests with the 1.10.0 RC artifacts. > > > > I did some minor tests with a toy python project (both py and py3). > > > > +1 for me (mostly Java, some python). > > > > If there is a blocker or another release candidate, AVRO-2849 (spec > > clarification) would be nice to include. > > > > All my best and thanks for the work, Ismael! > > > > Ryan > > > > > > On Thu, Jun 11, 2020 at 2:25 PM Daniel Kulp <[email protected]> wrote: > >> > >> +1 > >> > >> Been playing around with it a bit and it looks OK. > >> > >> Dan > >> > >> > >>> On Jun 11, 2020, at 4:53 AM, Ismaël Mejía <[email protected]> wrote: > >>> > >>> Kind reminder two days have passed, please check and vote! > >>> > >>> On Thu, Jun 11, 2020 at 2:29 AM Andy Le <[email protected]> wrote: > >>>> > >>>> Hey Ryan, > >>>> > >>>> Thank you for your clear explanation. > >>>> > >>>> +1 for me :D > >>>> > >>>> Thank you. > >>>> > >>>> > >>>> On 2020/06/10 14:39:10, Ryan Skraba <[email protected]> wrote: > >>>>> Hey Anh, a release requires 3 PMC votes, but "unofficial" opinions are > >>>>> welcome and taken into account. My opinion is unofficial too! > >>>>> > >>>>> Just reply with a +1 if you think the artifacts are good to go or -1 > >>>>> if you've found a bug, regression, security issue, or something that > >>>>> should block these artifacts from being the 1.10.0 release. > >>>>> > >>>>> For example, at my company, we have a lot of internal code and > >>>>> libraries that work with Avro 1.9.2, especially the generic Java APIs, > >>>>> so I'm going to compile with 1.10.0 and see if all the unit tests > >>>>> pass. > >>>>> > >>>>> If you're familiar with building any open source projects that use > >>>>> Avro, try bumping there and reporting back. You can go through the > >>>>> open JIRA and open PR and make recommendations if we missed something > >>>>> important. > >>>>> > >>>>> For example, I've gone through a lot of the JIRA and it looks like > >>>>> AVRO-2799 (a protobuf bug) is blocking a user. I'm looking to see if > >>>>> this is a regression or if it's an easy fix before I vote. > >>>>> > >>>>> I've verified the signatures and SHA512 of the artifacts, so that's > >>>>> good! > >>>>> > >>>>> All my best, Ryan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Jun 10, 2020 at 8:53 AM Anh Le <[email protected]> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> How can I make a vote on this? > >>>>>> > >>>>>> Best Regards. > >>>>>> > >>>>>> On Tue, 9 Jun 2020, 22:28 Ismaël Mejía, <[email protected]> wrote: > >>>>>> > >>>>>>> Hi everyone, > >>>>>>> > >>>>>>> I'd like to propose the following RC1 to be released as the official > >>>>>>> Apache > >>>>>>> Avro 1.10.0 release. > >>>>>>> > >>>>>>> The commit id is 1301a045396cf0e6ca7f12c9e4053b4f81237887 > >>>>>>> > >>>>>>> * This corresponds to the tag: release-1.10.0-rc1 > >>>>>>> * https://github.com/apache/avro/releases/tag/release-1.10.0-rc1 > >>>>>>> > >>>>>>> The release tarball, signature, and checksums are here (revision > >>>>>>> 37932): > >>>>>>> * https://dist.apache.org/repos/dist/dev/avro/avro-1.10.0-rc1/ > >>>>>>> > >>>>>>> You can find the KEYS file here: > >>>>>>> * https://dist.apache.org/repos/dist/dev/avro/KEYS > >>>>>>> > >>>>>>> Binary artifacts for Java are staged in Nexus here: > >>>>>>> * > >>>>>>> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.10.0/ > >>>>>>> > >>>>>>> This release includes 187 Jira issues (Thanks Ryan Skraba for the > >>>>>>> highlights) > >>>>>>> > >>>>>>> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.0 > >>>>>>> > >>>>>>> * Some interesting highlights (Thanks Ryan Skraba for the list): > >>>>>>> > >>>>>>> C# > >>>>>>> - [AVRO-2389] Add Avro serialization for POCO (Reflection) > >>>>>>> > >>>>>>> Java > >>>>>>> - [AVRO-2723] Automatically find defaults on POJO when using > >>>>>>> reflection. > >>>>>>> - [AVRO-2278] Now throws an exception when getting a non-existent > >>>>>>> field from a record > >>>>>>> - [AVRO-2335] Remove Joda Time library. > >>>>>>> - [AVRO-2438] Better support for URI and URL types. > >>>>>>> > >>>>>>> Perl > >>>>>>> - [AVRO-1461] Distribute as a CPAN module. > >>>>>>> > >>>>>>> PHP > >>>>>>> - [AVRO-2527] Update to PHP 7.x > >>>>>>> > >>>>>>> Python > >>>>>>> - [AVRO-2656] avro-python package is now the preferred python3 library > >>>>>>> and avro-python3 is prepared to be deprecated > >>>>>>> - [AVRO-2387] type checking added to python > >>>>>>> > >>>>>>> Ruby > >>>>>>> - [AVRO-1740] Support fingerprinting. > >>>>>>> - [AVRO-2535] Support enum defaults. > >>>>>>> - [AVRO-2545] Support aliases. > >>>>>>> > >>>>>>> * Upgrade dependencies to latest versions, including CVE fixes. > >>>>>>> * Multiple fixes and more... > >>>>>>> > >>>>>>> Please download, verify, and test. This vote will remain open for at > >>>>>>> least > >>>>>>> 72 hours. Given sufficient votes, I would like to close after noon UTC > >>>>>>> Friday, June 12th, 2020 > >>>>>>> > >>>>>>> [ ] +1 Release this as Apache Avro 1.10.0 > >>>>>>> [ ] +0 > >>>>>>> [ ] -1 Do not release this because... > >>>>>>> > >>>>>>> Best regards, > >>>>>>> Ismaël Mejía > >>>>>>> > >>>>> > >> > >> -- > >> Daniel Kulp > >> [email protected] <mailto:[email protected]> - http://dankulp.com/blog > >> <http://dankulp.com/blog> > >> Talend Community Coder - http://talend.com <http://coders.talend.com/> >
