Micah, I took a look at the proposed changes in #588 <https://github.com/apache/parquet-format/pull/588> and I don't think that's the right way forward (at least not entirely).
What I see is three independent changes that we shouldn't lump all together: 1. Parquet Format Semver 2. Addressing path_in_schema 3. Introducing a new magic number (PARX) I agree with adopting semver for parquet format releases (though there's still the question around how that aligns with the parquet footer version). For path_in_schema, I think we can leverage the current precedent of having a configurable flag to omit and produce an empty list. This would address the space concerns for those who want to opt out of compatibility, but we wouldn't default or change this to optional until there's a formal V3 release. I'm not convinced about the PARX magic number and extended footer capabilities bitset, but I don't think that's something we need to commit to just yet, so I don't feel there's a need to rush to a decision on that point. -Dan On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield <[email protected]> wrote: > To try to move the conversation forward I made two PRs: > > 1. parquet-format (https://github.com/apache/parquet-format/pull/588): > TL;DR; > - Changes language to use recommended specification version as the > mechanism for configuration. > - Commits to using SemVer for parquet-format releases going > forward (all forwards incompatible changes, including encodings, etc) will > bump the version number. > - Adds a proposal for a new PARX magic number that has a new fixed > length component to the file footer composed of (metadata_len, feature > bitmap, CRC for footer and 'PARX' trailer). This also unifies encrypted and > unencrypted parquet files. > > > 2. A POC in Rust on how this could be implemented ( > https://github.com/apache/arrow-rs/pull/10177), including its usage with > path_in_footer > > I think this meets the implicit requirements discussed in this thread. > Namely: > > 1. It allows users to think about versions in a canonical way for feature > enablement. > 2. Keep recommendations that won't push default versions too quickly. > 3. Allows for continuous and iterative releases of the specification. > 4. Allows flexibility for readers to determine at a granular level if they > can properly read the file (and forces a single forward incompatible change > so we aren't relying on guesswork for when a writer feature can be safely > enabled). > 5. Limits the need for new magic numbers past a single new value. > > Please let me know if I missed something. If we can gain consensus around > this, I can add a Java implementation so we can adopt the changes and vote > on them. > > Cheers, > Micah > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer <[email protected] > > > wrote: > > > I agree this is getting a bit too complicated, I feel like everyone here > > understands versions as does the wider community. Why not just start > there > > and add other techniques if that fails to work properly or be effective > for > > communication. > > > > > > I think we are better of just choosing something simple and going forward > > rather than deliberating, the worst thing that happens is that we have to > > make a different choice later. I’m not sure that’s worse than sitting on > > what we currently have and not being able to make progress on new > encodings > > or footers. > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield <[email protected]> > > wrote: > > > > > Thank you for the feedback Andrew. Practically speaking, I wonder if we > > > should have two separate notions of feature bundling: > > > > > > 1. Specification version (this would be primary and risks using > features > > > that aren't widely adopted). > > > 2. Presets - Gives users a different way of configuring things that > > allows > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell <[email protected]> > wrote: > > > > > > > Hi, > > > > > > > > This discussion and the proposals seem to have gotten very > complicated. > > > > People not on this mailing list or not doing regular development on > > > Parquet > > > > would probably benefit from simplicity. Most people are used to > version > > > > numbers without worrying about various types of compatibility -- > > readers > > > > can simply state "I can read version 3-7", for example. Users > > understand > > > > this. People writing files can also easily understand "I want to > write > > a > > > > version 6 file because version 6 supports feature X that I want." or > > "I > > > > want to write version 7 because it's the latest version." > > > > > > > > I don't really care about the details of a solution, but please keep > in > > > > mind that a more simple solution probably increases accessibility for > > the > > > > widest range of people. > > > > > > > > -- > > > > Andrew Bell > > > > [email protected] > > > > > > > > > >
