Hey everyone, What would be a good time to kick off the 1.13.1 release? Happy to run the release.
Cheers, Fokko Driesprong Op za 6 mei 2023 om 04:03 schreef Claire McGinty <[email protected] >: > Great, thank you! I realized that that PR does depend on PARQUET-2265 > <https://github.com/apache/parquet-mr/pull/1049/files> for the writes to > work -- it's a very minor PR bringing the default Avro write behavior in > line with Avro read behavior. It unlocks a new functionality--being able to > specify an Avro DataModel via a Configuration key--and has no user-facing > effect on any existing behavior. > > I created a cherry-pick PR containing both tickets here > <https://github.com/apache/parquet-mr/pull/1091> -- I tested my local > installation locally and on Hadoop, using Avro data containing logical > types, and confirmed that it's working as expected. > > Thanks, > Claire > > On Fri, May 5, 2023 at 6:27 PM Fokko Driesprong <[email protected]> wrote: > > > Hey all, > > > > Thanks for reviewing the PR on bringing back Hadoop 2.7.x support. I've > > created a backport <https://github.com/apache/parquet-mr/pull/1090> to > the > > 1.13.1 branch. > > > > Claire: Thanks for working on that. I looked through the PR and it looks > > well-tested, personally, I'm comfortable with cherry-picking this back to > > 1.13.1. If you can create a PR to the parquet-1.13.x branch we can take > it > > from there. > > > > Cheers, Fokko > > > > Op vr 5 mei 2023 om 15:14 schreef Claire McGinty < > > [email protected] > > >: > > > > > Hi all, > > > > > > I wanted to add another +1 to the requests for a 1.13.1 release—I just > > had > > > a PR merged to add built-in support for Avro logical types in > > > Avro{Read,Write}Support ( > https://github.com/apache/parquet-mr/pull/1078 > > ). > > > For context on this, my org, Spotify, is currently migrating many > > datasets > > > from Avro to Parquet, but since many of our Avro datasets use logical > > > types, it’s not working out of the box without custom config per > > > dataset/consumer. I know it’s unconventional to include new features > in a > > > patch release, but having this change available would really help us > > speed > > > up our Parquet adoption, plus the implementation is on a strictly > > > “best-effort” basis—any errors fetching logical types result in Parquet > > > reverting to its default read/write behavior. > > > > > > Thanks, > > > Claire > > > > > > > > > On 2023/05/01 06:56:37 Fokko Driesprong wrote: > > > > Hey everyone, > > > > > > > > I also got some pushback in Iceberg, so I've taken the time to revert > > the > > > > change to continue <https://github.com/apache/parquet-mr/pull/1084/> > > to > > > > support Hadoop <2.9. We now have two mechanisms to check if the > stream > > is > > > > byte buffer readable. First it will use the new mechanism (For Hadoop > > > 2.9+ > > > > and Hadoop 3.3+. Otherwise, it will fall back to the previous method. > > > > Please review and let me know what you think. Once in, I can backport > > > this > > > > to 1.13.1. > > > > > > > > Kind regards, > > > > Fokko Driesprong > > > > > > > > Op vr 28 apr 2023 om 10:06 schreef Fokko Driesprong < > [email protected] > > >: > > > > > > > > > And it is in Hive as well: https://github.com/apache/hive/pull/427 > > > > > > > > > > Kind regards, > > > > > Fokko > > > > > > > > > > Op wo 26 apr 2023 om 18:25 schreef Fokko Driesprong < > > [email protected] > > > >: > > > > > > > > > >> Hi Xinli, > > > > >> > > > > >> I know that some folks are waiting on running Apache Flink without > > > Hadoop > > > > >> (but using the parquet-mr library in Iceberg). Spark already > > upgraded > > > to > > > > >> Parquet 1.3.0, and I did some checks today with Hive, and I don't > > see > > > any > > > > >> blockers there. What's your understanding of a reasonable time > > window? > > > > >> Personally, I don't mind running a release, especially a patch > > release > > > > >> should be quite straightforward. > > > > >> > > > > >> Kind regards, > > > > >> Fokko Driesprong > > > > >> > > > > >> Op wo 26 apr 2023 om 04:37 schreef Xinli shang > > <[email protected] > > > >: > > > > >> > > > > >>> Hi Fokko, > > > > >>> > > > > >>> Thanks for volunteering to release 1.13.1! That would be great > and > > I > > > am > > > > >>> looking forward to you being the release manager for that. > > > > >>> > > > > >>> We can have the 1.13.1 release to add back the support old Hadoop > > > > >>> version, > > > > >>> but the question is should we release ASAP or wait for a > reasonable > > > time > > > > >>> window? The new version 1.13.0 is just released and I am not sure > > if > > > > >>> there > > > > >>> are more issues coming so that we can put together the fixes into > > > 1.13.1. > > > > >>> Is Iceberg urgently blocked on this? > > > > >>> > > > > >>> Xinli Shang > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Tue, Apr 25, 2023 at 6:51 PM Gang Wu <[email protected]> wrote: > > > > >>> > > > > >>> > That sounds good to me. > > > > >>> > > > > > >>> > I have just released 1.13.0, just let me know if you need > > anything > > > > >>> > on my end to make the next release. > > > > >>> > > > > > >>> > Best, > > > > >>> > Gang > > > > >>> > > > > > >>> > On Tue, Apr 25, 2023 at 10:31 PM Fokko Driesprong < > > > [email protected]> > > > > >>> > wrote: > > > > >>> > > > > > >>> > > Hey Gang, > > > > >>> > > > > > > >>> > > Thanks for the quick reply. I think 2.8.x is water under the > > > bridge, > > > > >>> but > > > > >>> > I > > > > >>> > > can be convinced otherwise. I also spend a few cycles to see > if > > > we > > > > >>> can > > > > >>> > get > > > > >>> > > compatibility with 2.7.3+, but it doesn't seem trivial > > > > >>> > > < > > > > >>> > > > https://github.com/apache/parquet-mr/pull/1075#issuecomment-1514518094 > > > > >>> > >. > > > > >>> > > As Gabor said on the ticket, it is fine to drop support for > > older > > > > >>> systems > > > > >>> > > from time to time. The public Hadoop 2.8 > > > > >>> > > <https://github.com/apache/hadoop/tree/branch-2.8> doesn't > > seem > > > to > > > > >>> get > > > > >>> > any > > > > >>> > > active updates. I don't fully agree with the ticket, you can > > > still > > > > >>> read > > > > >>> > > Parquet, but using an older version of the library. > > > > >>> > > > > > > >>> > > Kind regards, > > > > >>> > > Fokko Driesprong > > > > >>> > > > > > > >>> > > Op di 25 apr 2023 om 16:13 schreef Gang Wu <[email protected] > >: > > > > >>> > > > > > > >>> > > > Hi Fokko, > > > > >>> > > > > > > > >>> > > > There is an issue of the 1.13.0 release: > > > > >>> > > > https://issues.apache.org/jira/browse/PARQUET-2276. > > > > >>> > > > > > > > >>> > > > It seems that Hadoop 2.8.x is no longer supported after > > > 1.13.0. I > > > > >>> have > > > > >>> > > seen > > > > >>> > > > that > > > > >>> > > > you have added CI checks for Hadoop 2.9.x. Not sure if this > > is > > > a > > > > >>> > > > blocking issue. > > > > >>> > > > > > > > >>> > > > Best, > > > > >>> > > > Gang > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > On Tue, Apr 25, 2023 at 3:25 PM Fokko Driesprong < > > > [email protected] > > > > >>> > > > > > >>> > > wrote: > > > > >>> > > > > > > > >>> > > > > Hi all, > > > > >>> > > > > > > > > >>> > > > > I would like to discuss releasing Parquet 1.13.1. For > > > Iceberg we > > > > >>> ran > > > > >>> > > into > > > > >>> > > > > two things: > > > > >>> > > > > > > > > >>> > > > > - We noticed that support for Hadoop 2 was dropped. > > > Iceberg is > > > > >>> > still > > > > >>> > > > on > > > > >>> > > > > 2.7.3, and we're aware of the fact that has been > > released > > > in > > > > >>> > August > > > > >>> > > > > 2016. > > > > >>> > > > > The PR that I've created > > > > >>> > > > > <https://github.com/apache/parquet-mr/pull/1083/> > bumps > > > the > > > > >>> lower > > > > >>> > > > bound > > > > >>> > > > > to Hadoop 2.9.2. Which is also old, but if possible we > > > would > > > > >>> like > > > > >>> > to > > > > >>> > > > > cater > > > > >>> > > > > to the widest audience possible. > > > > >>> > > > > - At Iceberg we also have the Apache Flink > integration, > > > and > > > > >>> Flink > > > > >>> > is > > > > >>> > > > > able to run without Hadoop. This required some minor > > > changes > > > > >>> > (#1074 > > > > >>> > > > > <https://github.com/apache/parquet-mr/pull/1074>, > #1073 > > > > >>> > > > > <https://github.com/apache/parquet-mr/pull/1073>) > that > > > > >>> already > > > > >>> > have > > > > >>> > > > > been > > > > >>> > > > > backported. It would be awesome to get these out. > > > > >>> > > > > > > > > >>> > > > > My question is, after the release of 1.13.0 are there any > > > issues > > > > >>> that > > > > >>> > > > came > > > > >>> > > > > up, or anything that you would like to see being > released? > > > I'm > > > > >>> happy > > > > >>> > to > > > > >>> > > > > volunteer as a release manager for 1.13.1. Let us know! > > > > >>> > > > > > > > > >>> > > > > Kind regards, > > > > >>> > > > > Fokko > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> Xinli Shang > > > > >>> > > > > >> > > > > > > >
