I think the main issue is that we should be coordinating and aligning on what we want to include in a major semver update and not just forcing a new version because we made an incompatible change.
I'm strongly in favor of SemVer, but how we go about implementing it matters. Most of the in progress proposals (modular footer, non-contiguous pages, encodings, types) don't necessitate a major version update, so moving forward with this change just to cause a bump seems like the wrong way to get started on more coordinated releases. I would like to hear more input from others on the proposed updates to the magic number/footer. -Dan On Wed, Jun 24, 2026 at 12:50 PM Ed Seidl <[email protected]> wrote: > I'm largely in agreement with Micah, but want to address a few points. > > I proposed the path_in_schema changes in part to test our documented > process for dealing with forward incompatible changes. This process was > found to be wanting, so it's good we're addressing its deficiencies. I do > think that change (and other structural changes) should wait until we have > a clear mechanism in place to signal to old readers that these aren't the > bits you're looking for, and I feel changing the file magic is pretty much > the > only way to do this. > > I also agree with Micah that an empty list for path_in_schema is worse than > omitting the field (although using path_in_schema in the way Alkis > described > has its own issues, chief among them the fact that Parquet does not dictate > that column names are unique!?!). > > I agree that adopting SemVer and a mechanism to provide some informational > version string to users can be handled separately. > > Thanks, > Ed > > On 2026/06/24 17:50:57 Micah Kornfield wrote: > > Hi Dan, > > > > > I took a look at the proposed changes in #588 > > > <https://github.com/apache/parquet-format/pull/588> and I don't think > > > that's the right way forward (at least not entirely). > > > > > > Thank you for the review. I responded below with comments inline but > while > > I think I understand some of your reservations, I'm not actually sure > what > > you believe the right path forward is. Could you elaborate on what you > > believe the right path forward is? Specifically, where do we want to get > > to close this conversation? > > > > For path_in_schema, I think we can leverage the current precedent of > having > > > a configurable flag to omit and produce an empty list. > > > > > > I don't think we can emit an empty list here, it needs to be absent to > > prevent data corruption (so thrift parsers fail hard). Per Alkis's email > > on the the other thread [1] some readers are using this as a column > > resolution path, it's not clear how defensively they are coded and > omitting > > the list could cause data corruption (reading back all null values for > > columns that don't populate the field). We need something that breaks > hard > > for readers. I don't want to get into a state where ambiguously > > interpretable parquet files exist. > > > > > > > but we > > > wouldn't default or change this to optional until there's a formal V3 > > > release. > > > > > > I think this might be one of the points of different philosophies. I > think > > we should be releasing off of head and once a forwards incompatible > change > > is merged, the next release automatically gets a version bump. But it > > sounds like you might have different ideas here? > > > > I agree with adopting semver for parquet format releases > > > > > > Great, hopefully everyone else is ok with this. > > > > > > > (though there's > > > still the question around how that aligns with the parquet footer > version). > > > > > > I'm not convinced about the PARX magic number and extended footer > > > capabilities bitset, but I don't think that's something we need to > commit > > > to just yet, so I don't feel there's a need to rush to a decision on > that > > > point. > > > > > > For all intents and purposes the version field in the parquet footer > > version is useless today. There are several mainstream readers that > don't > > take any action on the version number, and there is at least one writer > > that allows it to be configurable or hard-codes it. Introducing a new > > magic number provides a mechanism to reset this. However, I think the > > version number should be treated as informational only (i.e. it shouldn't > > be validated by readers) and having a fine grained enumeration of > features > > a reader needs to understand is a better approach for the following > reasons: > > > > 1. It allows for readers to produce finer grained error messages for the > > options it doesn't support (for the 1 time cost of changing the magic > > number, which will produce "not a parquet file") > > 2. To maximize compatibility a writer only needs to record which forward > > incompatible features it used, it doesn't need to recompute the lowest > > possible major version the features fell into. For example, suppose a > > reader only knows how to read version 3. If new a writer chooses to > write > > version "4" but doesn't actually use a forward incompatible changes (e.g. > > Version 4 was created because of a new encoding scheme that didn't happen > > to be used), the writing code could still write "4" as that is what was > > requested but the reader could still read the produced file since there > are > > no forward incompatible bits set. > > > > Thanks, > > Micah > > > > > > [1] https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j > > > > > > > > > > > > On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> wrote: > > > > > Micah, > > > > > > I took a look at the proposed changes in #588 > > > <https://github.com/apache/parquet-format/pull/588> and I don't think > > > that's the right way forward (at least not entirely). > > > > > > What I see is three independent changes that we shouldn't lump all > > > together: > > > > > > 1. Parquet Format Semver > > > 2. Addressing path_in_schema > > > 3. Introducing a new magic number (PARX) > > > > > > I agree with adopting semver for parquet format releases (though > there's > > > still the question around how that aligns with the parquet footer > version). > > > > > > For path_in_schema, I think we can leverage the current precedent of > having > > > a configurable flag to omit and produce an empty list. This would > address > > > the space concerns for those who want to opt out of compatibility, but > we > > > wouldn't default or change this to optional until there's a formal V3 > > > release. > > > > > > I'm not convinced about the PARX magic number and extended footer > > > capabilities bitset, but I don't think that's something we need to > commit > > > to just yet, so I don't feel there's a need to rush to a decision on > that > > > point. > > > > > > -Dan > > > > > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield < > [email protected]> > > > wrote: > > > > > > > To try to move the conversation forward I made two PRs: > > > > > > > > 1. parquet-format (https://github.com/apache/parquet-format/pull/588 > ): > > > > TL;DR; > > > > - Changes language to use recommended specification version > as > > > the > > > > mechanism for configuration. > > > > - Commits to using SemVer for parquet-format releases going > > > > forward (all forwards incompatible changes, including encodings, etc) > > > will > > > > bump the version number. > > > > - Adds a proposal for a new PARX magic number that has a new > > > fixed > > > > length component to the file footer composed of (metadata_len, > feature > > > > bitmap, CRC for footer and 'PARX' trailer). This also unifies > encrypted > > > and > > > > unencrypted parquet files. > > > > > > > > > > > > 2. A POC in Rust on how this could be implemented ( > > > > https://github.com/apache/arrow-rs/pull/10177), including its usage > with > > > > path_in_footer > > > > > > > > I think this meets the implicit requirements discussed in this > thread. > > > > Namely: > > > > > > > > 1. It allows users to think about versions in a canonical way for > > > feature > > > > enablement. > > > > 2. Keep recommendations that won't push default versions too > quickly. > > > > 3. Allows for continuous and iterative releases of the > specification. > > > > 4. Allows flexibility for readers to determine at a granular level > if > > > they > > > > can properly read the file (and forces a single forward incompatible > > > change > > > > so we aren't relying on guesswork for when a writer feature can be > safely > > > > enabled). > > > > 5. Limits the need for new magic numbers past a single new value. > > > > > > > > Please let me know if I missed something. If we can gain consensus > > > around > > > > this, I can add a Java implementation so we can adopt the changes and > > > vote > > > > on them. > > > > > > > > Cheers, > > > > Micah > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > I agree this is getting a bit too complicated, I feel like everyone > > > here > > > > > understands versions as does the wider community. Why not just > start > > > > there > > > > > and add other techniques if that fails to work properly or be > effective > > > > for > > > > > communication. > > > > > > > > > > > > > > > I think we are better of just choosing something simple and going > > > forward > > > > > rather than deliberating, the worst thing that happens is that we > have > > > to > > > > > make a different choice later. I’m not sure that’s worse than > sitting > > > on > > > > > what we currently have and not being able to make progress on new > > > > encodings > > > > > or footers. > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Thank you for the feedback Andrew. Practically speaking, I > wonder if > > > we > > > > > > should have two separate notions of feature bundling: > > > > > > > > > > > > 1. Specification version (this would be primary and risks using > > > > features > > > > > > that aren't widely adopted). > > > > > > 2. Presets - Gives users a different way of configuring things > that > > > > > allows > > > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell <[email protected]> > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > This discussion and the proposals seem to have gotten very > > > > complicated. > > > > > > > People not on this mailing list or not doing regular > development on > > > > > > Parquet > > > > > > > would probably benefit from simplicity. Most people are used to > > > > version > > > > > > > numbers without worrying about various types of compatibility > -- > > > > > readers > > > > > > > can simply state "I can read version 3-7", for example. Users > > > > > understand > > > > > > > this. People writing files can also easily understand "I want > to > > > > write > > > > > a > > > > > > > version 6 file because version 6 supports feature X that I > want." > > > or > > > > > "I > > > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > > > > > I don't really care about the details of a solution, but please > > > keep > > > > in > > > > > > > mind that a more simple solution probably increases > accessibility > > > for > > > > > the > > > > > > > widest range of people. > > > > > > > > > > > > > > -- > > > > > > > Andrew Bell > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > >
