Hey Div, Thanks for raising the issue. Was just wrapping up the license stuff, but I think it makes sense to drive the nan-order to a conclusion before cutting another RC. Mostly because it affects correctness which we cannot jeopardize. Unfortunately I had to miss the Parquet sync (holiday period here in the EU, spending time with the fam), but looking forward to your separate thread.
Kind regards, Fokko On 2026/07/29 17:51:24 Divjot Arora via dev wrote: > Hi folks, > > Should we resolve > https://github.com/apache/parquet-java/pull/3393#issuecomment-5105352662 > before releasing parquet-java? My understanding of the current state is > that emitting nan_counts but using TYPE_DEFINED_ORDER could result in > legacy readers seeing finite min/max stats and ignoring nan_counts. In this > case, such readers could use the min/max stats for pruning and get wrong > results. However, using the new IEEE sort order has its own risks as there > are likely legacy readers that cannot parse unknown Thrift enum values. > > This was also discussed in today's Parquet sync meeting and I will start a > separate thread to discuss the right way forward here. Right now, it seems > we have a disconnect between parquet-format, where adding the new sort > order was considered a forward-compatible change, and reference > implementations, where we seem to think that the new orders can only be > used in the next major version bump. > > -- Div > > On Sun, Jul 26, 2026 at 11:45 PM Kevin Liu <[email protected]> wrote: > > > Here's a tracking issue for all the LICENSE / NOTICE fixes: "Ensure all > > published JARs contain correct LICENSE and NOTICE files" #3687 ( > > https://github.com/apache/parquet-java/issues/3687) > > > > This will > > - include LICENSE and NOTICE files in every jar > > - add CI to validate LICENSE and NOTICE files are included in every jar > > - update LICENSE and NOTICE for jars that bundle 3rd party code (this is an > > ASF release requirement) > > > > I just went through the process to update LICENSE and NOTICE files for > > iceberg-java, so I'm still fairly familiar with the process. I would love > > to have someone double check though :) > > > > > > > > > > On Sun, Jul 26, 2026 at 1:13 PM Fokko Driesprong <[email protected]> wrote: > > > > > Thanks Kevin, let me look into that. > > > > > > I also found a behavioral change, which I believe is problematic. PR 3575 > > > changes how we emit statistics: > > > https://github.com/apache/parquet-java/pull/3575. When there is one or > > > more row-groups that don't have bounds we don't produce statistics at the > > > file level. The PR changes this by always emitting the null-values (which > > > will automatically produce file-level stats). In Iceberg we rely on the > > > fact when statistics are null when one or more row-groups don't have > > stats: > > > https://github.com/apache/iceberg/pull/17321#discussion_r3641772062. I > > > think this behavioral change cause downstream issues which I think we > > > should avoid as it might introduce correctness issues. > > > > > > I took the liberty of creating a PR to revert that specific change: > > > https://github.com/apache/parquet-java/pull/3688. Please chime in to let > > > me know your thoughts. > > > > > > Kind regards, > > > Fokko > > > > > > On 2026/07/22 21:43:03 Kevin Liu wrote: > > > > I found a potential issue with the license and notice files that might > > > be a > > > > release blocker. > > > > > > > > Many of the jars lack license files, notice files, or both. > > > > Out of 59 staged JARs: > > > > - 40 lack both files > > > > - 2 lack LICENSE > > > > - 10 lack NOTICE > > > > The remaining 7 contain both, but their contents might be incomplete or > > > > outdated. > > > > > > > > Here's the script I used to verify and the result from the audit: > > > > > > > > > https://gist.github.com/kevinjqliu/2f899c085abb2716b13c220d4f52a677?permalink_comment_id=6271325#gistcomment-6271325 > > > > > > > > I also found that the NOTICE file's copyright year is out of date ( > > > > > > > > > https://github.com/apache/parquet-java/blob/83c2c80d49c57b1ae180df45229ee01dbceddc1e/NOTICE#L3 > > > > ) > > > > > > > > This is my first time verifying a parquet-java release, so please let > > me > > > > know if there's anything I'm missing. > > > > > > > > Best, > > > > Kevin Liu > > > > > > > > On Wed, Jul 22, 2026 at 1:30 PM Steve Loughran <[email protected]> > > > wrote: > > > > > > > > > +1 non-binding. > > > > > > > > > > I only audited the tarball and the maven artifacts, not the code; > > > leaving > > > > > that to others. I was answering the question "did the release-manager > > > place > > > > > anything malicious in the nexus artifacts", which is something you > > > should > > > > > be doing for all releases these days. > > > > > > > > > > Verified the RC1 source tarball and staged Maven artifacts: > > > > > * Source tarball SHA-512 OK; GPG signature good (Fokko Driesprong, > > key > > > 2F37 > > > > > FE05 5E54 7EDD ACD9 A633 B13C 6981 6C5E 13EF). Signed Fokko's key > > BTW, > > > as > > > > > nobody else has. > > > > > * Extracted source matches tag apache-parquet-1.18.0-rc1 exactly > > > > > * All 15 staged modules (104 artifacts): SHA-1/MD5 checksums match > > and > > > > > every GPG signature is good > > > > > * Rebuilt from the tag with JDK 11 (mvn clean package): > > > parquet-encoding > > > > > and parquet-jackson jars are byte-identical to the release; the rest > > > differ > > > > > only in bytecode traceable to a javac string-concat idiom > > > (String.valueOf + > > > > > makeConcatWithConstants). > > > > > * No structural differences (no added/removed classes, methods, or > > > fields) > > > > > and no new suspicious API calls in any jar. > > > > > > > > > > For anyone who wants to do that jar comparison, Auditor: > > > > > https://github.com/steveloughran/auditor > > > > > This disassembles the binary and looks at the bytecodes, so doesn't > > > require > > > > > a byte for byte match. Give it to claude to validate (as done here) > > and > > > > > it'll analyse why there there's a mismatch. > > > > > > > > > > steve > > > > > > > > > > > > > > > On Mon, 20 Jul 2026 at 15:30, Fokko Driesprong <[email protected] > > > > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I propose the following RC to be released as official Apache > > Parquet > > > > > > 1.18.0 release. > > > > > > > > > > > > - The commit id is [5e2ad0969c7f411613671c5ef054d399995de5f9]( > > > > > > > > > > > > > > > > https://github.com/apache/parquet-java/tree/5e2ad0969c7f411613671c5ef054d399995de5f9 > > > > > > ) > > > > > > - This corresponds to the tag: [apache-parquet-1.18.0-rc1]( > > > > > > > > > > > > > > > > https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.18.0-rc1 > > > > > > ) > > > > > > > > > > > > The release tarball, signature, and checksums are here: > > > > > > > > > > > > - > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/parquet/apache-parquet-1.18.0-rc1/ > > > > > > > > > > > > You can find the KEYS file here: > > > > > > > > > > > > - https://downloads.apache.org/parquet/KEYS > > > > > > > > > > > > You can find the changelog here: > > > > > > > > > > > > - > > > > > > > > > > > > > > > > https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.18.0-rc1 > > > > > > > > > > > > Binary artifacts are staged in Nexus here: > > > > > > > > > > > > - > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/parquet/ > > > > > > > > > > > > Please download, verify, and test. > > > > > > > > > > > > Please vote in the next 72 hours. > > > > > > > > > > > > [ ] +1 Release this as Apache Parquet 1.18.0 > > > > > > [ ] +0 > > > > > > [ ] -1 Do not release this because... > > > > > > > > > > > > Kind regards, > > > > > > Fokko Driesprong > > > > > > > > > > > > > > >
