Hi Dan, > I took a look at the proposed changes in #588 > <https://github.com/apache/parquet-format/pull/588> and I don't think > that's the right way forward (at least not entirely).
Thank you for the review. I responded below with comments inline but while I think I understand some of your reservations, I'm not actually sure what you believe the right path forward is. Could you elaborate on what you believe the right path forward is? Specifically, where do we want to get to close this conversation? For path_in_schema, I think we can leverage the current precedent of having > a configurable flag to omit and produce an empty list. I don't think we can emit an empty list here, it needs to be absent to prevent data corruption (so thrift parsers fail hard). Per Alkis's email on the the other thread [1] some readers are using this as a column resolution path, it's not clear how defensively they are coded and omitting the list could cause data corruption (reading back all null values for columns that don't populate the field). We need something that breaks hard for readers. I don't want to get into a state where ambiguously interpretable parquet files exist. > but we > wouldn't default or change this to optional until there's a formal V3 > release. I think this might be one of the points of different philosophies. I think we should be releasing off of head and once a forwards incompatible change is merged, the next release automatically gets a version bump. But it sounds like you might have different ideas here? I agree with adopting semver for parquet format releases Great, hopefully everyone else is ok with this. > (though there's > still the question around how that aligns with the parquet footer version). I'm not convinced about the PARX magic number and extended footer > capabilities bitset, but I don't think that's something we need to commit > to just yet, so I don't feel there's a need to rush to a decision on that > point. For all intents and purposes the version field in the parquet footer version is useless today. There are several mainstream readers that don't take any action on the version number, and there is at least one writer that allows it to be configurable or hard-codes it. Introducing a new magic number provides a mechanism to reset this. However, I think the version number should be treated as informational only (i.e. it shouldn't be validated by readers) and having a fine grained enumeration of features a reader needs to understand is a better approach for the following reasons: 1. It allows for readers to produce finer grained error messages for the options it doesn't support (for the 1 time cost of changing the magic number, which will produce "not a parquet file") 2. To maximize compatibility a writer only needs to record which forward incompatible features it used, it doesn't need to recompute the lowest possible major version the features fell into. For example, suppose a reader only knows how to read version 3. If new a writer chooses to write version "4" but doesn't actually use a forward incompatible changes (e.g. Version 4 was created because of a new encoding scheme that didn't happen to be used), the writing code could still write "4" as that is what was requested but the reader could still read the produced file since there are no forward incompatible bits set. Thanks, Micah [1] https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> wrote: > Micah, > > I took a look at the proposed changes in #588 > <https://github.com/apache/parquet-format/pull/588> and I don't think > that's the right way forward (at least not entirely). > > What I see is three independent changes that we shouldn't lump all > together: > > 1. Parquet Format Semver > 2. Addressing path_in_schema > 3. Introducing a new magic number (PARX) > > I agree with adopting semver for parquet format releases (though there's > still the question around how that aligns with the parquet footer version). > > For path_in_schema, I think we can leverage the current precedent of having > a configurable flag to omit and produce an empty list. This would address > the space concerns for those who want to opt out of compatibility, but we > wouldn't default or change this to optional until there's a formal V3 > release. > > I'm not convinced about the PARX magic number and extended footer > capabilities bitset, but I don't think that's something we need to commit > to just yet, so I don't feel there's a need to rush to a decision on that > point. > > -Dan > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield <[email protected]> > wrote: > > > To try to move the conversation forward I made two PRs: > > > > 1. parquet-format (https://github.com/apache/parquet-format/pull/588): > > TL;DR; > > - Changes language to use recommended specification version as > the > > mechanism for configuration. > > - Commits to using SemVer for parquet-format releases going > > forward (all forwards incompatible changes, including encodings, etc) > will > > bump the version number. > > - Adds a proposal for a new PARX magic number that has a new > fixed > > length component to the file footer composed of (metadata_len, feature > > bitmap, CRC for footer and 'PARX' trailer). This also unifies encrypted > and > > unencrypted parquet files. > > > > > > 2. A POC in Rust on how this could be implemented ( > > https://github.com/apache/arrow-rs/pull/10177), including its usage with > > path_in_footer > > > > I think this meets the implicit requirements discussed in this thread. > > Namely: > > > > 1. It allows users to think about versions in a canonical way for > feature > > enablement. > > 2. Keep recommendations that won't push default versions too quickly. > > 3. Allows for continuous and iterative releases of the specification. > > 4. Allows flexibility for readers to determine at a granular level if > they > > can properly read the file (and forces a single forward incompatible > change > > so we aren't relying on guesswork for when a writer feature can be safely > > enabled). > > 5. Limits the need for new magic numbers past a single new value. > > > > Please let me know if I missed something. If we can gain consensus > around > > this, I can add a Java implementation so we can adopt the changes and > vote > > on them. > > > > Cheers, > > Micah > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > [email protected] > > > > > wrote: > > > > > I agree this is getting a bit too complicated, I feel like everyone > here > > > understands versions as does the wider community. Why not just start > > there > > > and add other techniques if that fails to work properly or be effective > > for > > > communication. > > > > > > > > > I think we are better of just choosing something simple and going > forward > > > rather than deliberating, the worst thing that happens is that we have > to > > > make a different choice later. I’m not sure that’s worse than sitting > on > > > what we currently have and not being able to make progress on new > > encodings > > > or footers. > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield <[email protected] > > > > > wrote: > > > > > > > Thank you for the feedback Andrew. Practically speaking, I wonder if > we > > > > should have two separate notions of feature bundling: > > > > > > > > 1. Specification version (this would be primary and risks using > > features > > > > that aren't widely adopted). > > > > 2. Presets - Gives users a different way of configuring things that > > > allows > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell <[email protected]> > > wrote: > > > > > > > > > Hi, > > > > > > > > > > This discussion and the proposals seem to have gotten very > > complicated. > > > > > People not on this mailing list or not doing regular development on > > > > Parquet > > > > > would probably benefit from simplicity. Most people are used to > > version > > > > > numbers without worrying about various types of compatibility -- > > > readers > > > > > can simply state "I can read version 3-7", for example. Users > > > understand > > > > > this. People writing files can also easily understand "I want to > > write > > > a > > > > > version 6 file because version 6 supports feature X that I want." > or > > > "I > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > I don't really care about the details of a solution, but please > keep > > in > > > > > mind that a more simple solution probably increases accessibility > for > > > the > > > > > widest range of people. > > > > > > > > > > -- > > > > > Andrew Bell > > > > > [email protected] > > > > > > > > > > > > > > >
