Sorry I could not answer in time. Hectic week. I understand perfectly the decision of going ahead with the Parquet release, it is not wise some times to add fixes/improvements in the last minute. Of course I would have preferred to have the dependencies included but Gador arguments are solid so hopefully there won't be any issue.
Thanks for working on the release Gador and for the patient discussion. Regards, Ismaël On Thu, Mar 25, 2021 at 6:49 PM Gidon Gershinsky <[email protected]> wrote: > > Great news!!! > And thanks Gabor and Xinli for handling the release process! > > Cheers, Gidon > > > On Thu, Mar 25, 2021 at 7:01 PM Xinli shang <[email protected]> wrote: > > > Thanks everybody for the verification and special thanks to all the > > contributors to this release! This release includes awesome features and > > improvements. We look forward to the industry's adoption! > > > > On Thu, Mar 25, 2021 at 3:35 AM Gabor Szadovszky <[email protected]> wrote: > > > > > The vote to release 1.12.0 RC4 as Apache Parquet MR 1.12.0 is PASSED with > > > the required three +1 binding votes and two +1 non-binding votes. (There > > > were no -1 or 0 votes.) > > > Thank you all who verified and voted! > > > > > > I'm going forward with the release process soon. > > > > > > On Thu, Mar 25, 2021 at 1:26 AM Julien Le Dem <[email protected]> > > wrote: > > > > > > > +1 (binding) > > > > I verified the signature and built from source. > > > > All tests pass. > > > > Looks good. > > > > > > > > On Wed, Mar 24, 2021 at 2:07 AM Gabor Szadovszky <[email protected]> > > > wrote: > > > > > > > > > I currently have the feeling that the Avro/Jackson related issue has > > > been > > > > > discussed and the community agrees on moving forward with this RC as > > is > > > > > (without upgrading the Avro and the Jackson dependencies). > > > > > So, I'm giving my +1 (binding) vote. > > > > > > > > > > On Tue, Mar 23, 2021 at 9:28 PM Aaron Niskode-Dossett > > > > > <[email protected]> wrote: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > - cloned the 1.12.0-rc-4 tag from github > > > > > > - compiled jars locally and all tests passed > > > > > > - used the 1.12.0 jars as dependencies for a local application that > > > > > streams > > > > > > data into protobuf-parquet files > > > > > > - confirmed data is correct and can be read with parquet-tools > > > compiled > > > > > > from parquet 1.11.1 > > > > > > > > > > > > On Tue, Mar 23, 2021 at 10:47 AM Xinli shang > > <[email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Let's discuss it in today's community sync meeting. > > > > > > > > > > > > > > On Tue, Mar 23, 2021 at 8:37 AM Aaron Niskode-Dossett > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > Gabor and Ismaël, thank you both for the very clear > > explanations > > > of > > > > > > > what's > > > > > > > > going on. > > > > > > > > > > > > > > > > Based on Gabor's description of avro compatibility I would be > > +1 > > > > > > > > (non-binding) for the current RC. > > > > > > > > > > > > > > > > On Tue, Mar 23, 2021 at 4:36 AM Gabor Szadovszky < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks, Ismaël for the explanation. I have a couple of notes > > > > about > > > > > > your > > > > > > > > > concerns. > > > > > > > > > > > > > > > > > > - Parquet 1.12.0 as per the semantic versioning is not a > > major > > > > but > > > > > a > > > > > > > > minor > > > > > > > > > release. (It is different from the Avro versioning strategy > > > where > > > > > the > > > > > > > > > second version number means major version changes.) > > > > > > > > > - The jackson dependency is shaded in the parquet jars so the > > > > > > > > > synchronization of the version is not needed (and not even > > > > > possible). > > > > > > > > > - Using the latest Avro version makes sense but if we do not > > > use > > > > it > > > > > > for > > > > > > > > the > > > > > > > > > current release it should not cause any issues in our > > clients. > > > > > Let's > > > > > > > > check > > > > > > > > > the following example. We upgrade to the latest 1.10.2 Avro > > > > release > > > > > > in > > > > > > > > > parquet then release it under 1.12.0. Later on Avro creates a > > > new > > > > > > > release > > > > > > > > > (e.g. 1.10.3 or even 1.11.0) while Parquet does not. In this > > > case > > > > > our > > > > > > > > > clients need to upgrade Avro without Parquet. If it is a > > major > > > > Avro > > > > > > > > release > > > > > > > > > it might occur that the Parquet code has to be updated but > > > > usually > > > > > it > > > > > > > is > > > > > > > > > not the case. (The last time we've had to change production > > > code > > > > > for > > > > > > an > > > > > > > > > Avro upgrade was from 1.7.6 to 1.8.0.) I think our clients > > > should > > > > > be > > > > > > > able > > > > > > > > > to upgrade Avro independently from Parquet and vice versa > > > (until > > > > > > there > > > > > > > > are > > > > > > > > > incompatibility issues). I would even change Parquet's Avro > > > > > > dependency > > > > > > > to > > > > > > > > > "provided" but that might be a breaking change and clearly > > > won't > > > > do > > > > > > it > > > > > > > > just > > > > > > > > > before the release. > > > > > > > > > > > > > > > > > > What do you think? Anyone have a strong opinion about this > > > topic? > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Gabor > > > > > > > > > > > > > > > > > > On Mon, Mar 22, 2021 at 6:31 PM Ismaël Mejía < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sure. The Avro upgrade feature/API wise is minor for > > Parquet, > > > > so > > > > > > the > > > > > > > > > > possibility of adding a regression is really REALLY minor. > > > The > > > > > > hidden > > > > > > > > > issue > > > > > > > > > > is the new transitive dependencies introduced by Avro, > > > > concretely > > > > > > > > Jackson > > > > > > > > > > 2.12.2. > > > > > > > > > > > > > > > > > > > > Since Parquet 1.12.0 is a major version it is probably a > > good > > > > > > moment > > > > > > > to > > > > > > > > > > upgrade Jackson too that's why I opened [1] (already > > merged). > > > > In > > > > > > > > > particular > > > > > > > > > > now that Spark merged support for both Avro 1.10.2 [1] and > > > > > Jackson > > > > > > > > 2.12.2 > > > > > > > > > > [2] for the upcoming 3.2.0 release, so now Spark can easily > > > > bring > > > > > > > > > upgraded > > > > > > > > > > Parquet too with all the dependencies well aligned. This of > > > > > course > > > > > > is > > > > > > > > > not a > > > > > > > > > > blocker for the release or for other downstream projects > > but > > > it > > > > > > might > > > > > > > > > help > > > > > > > > > > to make their life better because they will have less > > > > dependency > > > > > > > > > alignment > > > > > > > > > > issues to battle. > > > > > > > > > > > > > > > > > > > > Ismaël > > > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/parquet-mr/pull/883 > > > > > > > > > > [2] https://github.com/apache/spark/pull/31866 > > > > > > > > > > [3] https://github.com/apache/spark/pull/31878 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 22, 2021, 3:37 PM Xinli shang > > > > > <[email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Ismaël, > > > > > > > > > > > > > > > > > > > > > > Can you explain a little bit more on if we don't upgrade > > in > > > > > this > > > > > > > > > release, > > > > > > > > > > > what could be the worst-case scenario for the ecosystem? > > > The > > > > > > > > > last-minute > > > > > > > > > > > upgrading seems a rush to me but I would like to hear > > what > > > > are > > > > > > the > > > > > > > > > impact > > > > > > > > > > > if we don't. As Gabor mentioned, this should not be a > > > > > > > show-stopper. > > > > > > > > > > > > > > > > > > > > > > Xinli > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 22, 2021 at 1:55 AM Gabor Szadovszky < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot for the ones who checked the release and > > > > voted. > > > > > We > > > > > > > > have > > > > > > > > > > two > > > > > > > > > > > +1 > > > > > > > > > > > > votes (1 binding) so far. > > > > > > > > > > > > > > > > > > > > > > > > About the Avro + Jackson upgrade. I would like to hear > > > your > > > > > > > > opinions > > > > > > > > > > > about > > > > > > > > > > > > the current vote. Do you think we shall fail the > > current > > > > vote > > > > > > to > > > > > > > > > > upgrade > > > > > > > > > > > > these versions for the next RC or you accept this RC as > > > is > > > > > for > > > > > > > the > > > > > > > > > next > > > > > > > > > > > > release. (NOTE: as per semantic versioning this release > > > in > > > > > not > > > > > > a > > > > > > > > > major > > > > > > > > > > > but > > > > > > > > > > > > a minor release of parquet-mr.) > > > > > > > > > > > > > > > > > > > > > > > > I give a +0 (binding) for now until we have a clear > > view > > > > > about > > > > > > > the > > > > > > > > > > > previous > > > > > > > > > > > > topic. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Gabor > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 20, 2021 at 12:44 PM Grisha Weintraub < > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > Verified hash, signature and successful build. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 17, 2021 at 11:22 AM Gabor Szadovszky < > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I propose the following RC to be released as the > > > > official > > > > > > > > Apache > > > > > > > > > > > > Parquet > > > > > > > > > > > > > > 1.12.0 release. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The commit id is > > > > db75a6815f2ba1d1ee89d1a90aeb296f1f3a8f20 > > > > > > > > > > > > > > * This corresponds to the tag: > > > > apache-parquet-1.12.0-rc4 > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-mr/tree/db75a6815f2ba1d1ee89d1a90aeb296f1f3a8f20 > > > > > > > > > > > > > > > > > > > > > > > > > > > > The release tarball, signature, and checksums are > > > here: > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.12.0-rc4 > > > > > > > > > > > > > > > > > > > > > > > > > > > > You can find the KEYS file here: > > > > > > > > > > > > > > * https://downloads.apache.org/parquet/KEYS > > > > > > > > > > > > > > > > > > > > > > > > > > > > Binary artifacts are staged in Nexus here: > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/parquet/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > This release includes the features Parquet Modular > > > > > > Encryption > > > > > > > > and > > > > > > > > > > > > Parquet > > > > > > > > > > > > > > Bloom Filter. See details at: > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.0-rc4/CHANGES.md > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please download, verify, and test. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please vote in the next 72 hours. > > > > > > > > > > > > > > > > > > > > > > > > > > > > [ ] +1 Release this as Apache Parquet 1.12.0 > > > > > > > > > > > > > > [ ] +0 > > > > > > > > > > > > > > [ ] -1 Do not release this because... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Xinli Shang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Aaron Niskode-Dossett, Data Engineering -- Etsy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Xinli Shang > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Aaron Niskode-Dossett, Data Engineering -- Etsy > > > > > > > > > > > > > > > > > > > > > > > > -- > > Xinli Shang > >
