I'm largely in agreement with Micah, but want to address a few points. I proposed the path_in_schema changes in part to test our documented process for dealing with forward incompatible changes. This process was found to be wanting, so it's good we're addressing its deficiencies. I do think that change (and other structural changes) should wait until we have a clear mechanism in place to signal to old readers that these aren't the bits you're looking for, and I feel changing the file magic is pretty much the only way to do this.
I also agree with Micah that an empty list for path_in_schema is worse than omitting the field (although using path_in_schema in the way Alkis described has its own issues, chief among them the fact that Parquet does not dictate that column names are unique!?!). I agree that adopting SemVer and a mechanism to provide some informational version string to users can be handled separately. Thanks, Ed On 2026/06/24 17:50:57 Micah Kornfield wrote: > Hi Dan, > > > I took a look at the proposed changes in #588 > > <https://github.com/apache/parquet-format/pull/588> and I don't think > > that's the right way forward (at least not entirely). > > > Thank you for the review. I responded below with comments inline but while > I think I understand some of your reservations, I'm not actually sure what > you believe the right path forward is. Could you elaborate on what you > believe the right path forward is? Specifically, where do we want to get > to close this conversation? > > For path_in_schema, I think we can leverage the current precedent of having > > a configurable flag to omit and produce an empty list. > > > I don't think we can emit an empty list here, it needs to be absent to > prevent data corruption (so thrift parsers fail hard). Per Alkis's email > on the the other thread [1] some readers are using this as a column > resolution path, it's not clear how defensively they are coded and omitting > the list could cause data corruption (reading back all null values for > columns that don't populate the field). We need something that breaks hard > for readers. I don't want to get into a state where ambiguously > interpretable parquet files exist. > > > > but we > > wouldn't default or change this to optional until there's a formal V3 > > release. > > > I think this might be one of the points of different philosophies. I think > we should be releasing off of head and once a forwards incompatible change > is merged, the next release automatically gets a version bump. But it > sounds like you might have different ideas here? > > I agree with adopting semver for parquet format releases > > > Great, hopefully everyone else is ok with this. > > > > (though there's > > still the question around how that aligns with the parquet footer version). > > > I'm not convinced about the PARX magic number and extended footer > > capabilities bitset, but I don't think that's something we need to commit > > to just yet, so I don't feel there's a need to rush to a decision on that > > point. > > > For all intents and purposes the version field in the parquet footer > version is useless today. There are several mainstream readers that don't > take any action on the version number, and there is at least one writer > that allows it to be configurable or hard-codes it. Introducing a new > magic number provides a mechanism to reset this. However, I think the > version number should be treated as informational only (i.e. it shouldn't > be validated by readers) and having a fine grained enumeration of features > a reader needs to understand is a better approach for the following reasons: > > 1. It allows for readers to produce finer grained error messages for the > options it doesn't support (for the 1 time cost of changing the magic > number, which will produce "not a parquet file") > 2. To maximize compatibility a writer only needs to record which forward > incompatible features it used, it doesn't need to recompute the lowest > possible major version the features fell into. For example, suppose a > reader only knows how to read version 3. If new a writer chooses to write > version "4" but doesn't actually use a forward incompatible changes (e.g. > Version 4 was created because of a new encoding scheme that didn't happen > to be used), the writing code could still write "4" as that is what was > requested but the reader could still read the produced file since there are > no forward incompatible bits set. > > Thanks, > Micah > > > [1] https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j > > > > > > On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> wrote: > > > Micah, > > > > I took a look at the proposed changes in #588 > > <https://github.com/apache/parquet-format/pull/588> and I don't think > > that's the right way forward (at least not entirely). > > > > What I see is three independent changes that we shouldn't lump all > > together: > > > > 1. Parquet Format Semver > > 2. Addressing path_in_schema > > 3. Introducing a new magic number (PARX) > > > > I agree with adopting semver for parquet format releases (though there's > > still the question around how that aligns with the parquet footer version). > > > > For path_in_schema, I think we can leverage the current precedent of having > > a configurable flag to omit and produce an empty list. This would address > > the space concerns for those who want to opt out of compatibility, but we > > wouldn't default or change this to optional until there's a formal V3 > > release. > > > > I'm not convinced about the PARX magic number and extended footer > > capabilities bitset, but I don't think that's something we need to commit > > to just yet, so I don't feel there's a need to rush to a decision on that > > point. > > > > -Dan > > > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield <[email protected]> > > wrote: > > > > > To try to move the conversation forward I made two PRs: > > > > > > 1. parquet-format (https://github.com/apache/parquet-format/pull/588): > > > TL;DR; > > > - Changes language to use recommended specification version as > > the > > > mechanism for configuration. > > > - Commits to using SemVer for parquet-format releases going > > > forward (all forwards incompatible changes, including encodings, etc) > > will > > > bump the version number. > > > - Adds a proposal for a new PARX magic number that has a new > > fixed > > > length component to the file footer composed of (metadata_len, feature > > > bitmap, CRC for footer and 'PARX' trailer). This also unifies encrypted > > and > > > unencrypted parquet files. > > > > > > > > > 2. A POC in Rust on how this could be implemented ( > > > https://github.com/apache/arrow-rs/pull/10177), including its usage with > > > path_in_footer > > > > > > I think this meets the implicit requirements discussed in this thread. > > > Namely: > > > > > > 1. It allows users to think about versions in a canonical way for > > feature > > > enablement. > > > 2. Keep recommendations that won't push default versions too quickly. > > > 3. Allows for continuous and iterative releases of the specification. > > > 4. Allows flexibility for readers to determine at a granular level if > > they > > > can properly read the file (and forces a single forward incompatible > > change > > > so we aren't relying on guesswork for when a writer feature can be safely > > > enabled). > > > 5. Limits the need for new magic numbers past a single new value. > > > > > > Please let me know if I missed something. If we can gain consensus > > around > > > this, I can add a Java implementation so we can adopt the changes and > > vote > > > on them. > > > > > > Cheers, > > > Micah > > > > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > > [email protected] > > > > > > > wrote: > > > > > > > I agree this is getting a bit too complicated, I feel like everyone > > here > > > > understands versions as does the wider community. Why not just start > > > there > > > > and add other techniques if that fails to work properly or be effective > > > for > > > > communication. > > > > > > > > > > > > I think we are better of just choosing something simple and going > > forward > > > > rather than deliberating, the worst thing that happens is that we have > > to > > > > make a different choice later. I’m not sure that’s worse than sitting > > on > > > > what we currently have and not being able to make progress on new > > > encodings > > > > or footers. > > > > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield <[email protected] > > > > > > > wrote: > > > > > > > > > Thank you for the feedback Andrew. Practically speaking, I wonder if > > we > > > > > should have two separate notions of feature bundling: > > > > > > > > > > 1. Specification version (this would be primary and risks using > > > features > > > > > that aren't widely adopted). > > > > > 2. Presets - Gives users a different way of configuring things that > > > > allows > > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell <[email protected]> > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > This discussion and the proposals seem to have gotten very > > > complicated. > > > > > > People not on this mailing list or not doing regular development on > > > > > Parquet > > > > > > would probably benefit from simplicity. Most people are used to > > > version > > > > > > numbers without worrying about various types of compatibility -- > > > > readers > > > > > > can simply state "I can read version 3-7", for example. Users > > > > understand > > > > > > this. People writing files can also easily understand "I want to > > > write > > > > a > > > > > > version 6 file because version 6 supports feature X that I want." > > or > > > > "I > > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > > > I don't really care about the details of a solution, but please > > keep > > > in > > > > > > mind that a more simple solution probably increases accessibility > > for > > > > the > > > > > > widest range of people. > > > > > > > > > > > > -- > > > > > > Andrew Bell > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > >
