Hi Dan, Responses inline.
I think the main issue is that we should be coordinating and aligning on > what we want to include in a major semver update and not just forcing a new > version because we made an incompatible change. To have a practical discussion I think it would help to post a PR with a proposal to CONTRIBUTING.md on how such a process would work so we can evaluate the difference with the what is currently in contributing.md, the strawman posted earlier in the thread ( https://github.com/apache/parquet-format/pull/588) and how you are thinking about the problem. One of my biggest concerns is having mechanisms in place to avoid falling into the "V2" trap. > Most of the in progress proposals (modular footer, non-contiguous pages, > encodings, types) don't necessitate a major version update, Non-contiguous pages and encodings are forward-incompatible changes and in my mind would definitely necessitate a major version bump. The modular footer, depending on implementation, would also likely fall into a forward-incompatible change category. I agree new types would not require a major version upgrade, as they are already classified as forward-compatible. Since there seems to be some misalignment here, covering how you are thinking about this working as part of the process in a PR proposal would be valuable to further the discussion. so moving > forward with this change just to cause a bump seems like the wrong way to > get started on more coordinated releases. While Ed mentioned this was a test-case for a forward incompatible change, I at least did not see this as the primary focus of the proposal. My understanding is that this change offers real practical benefits for reducing the footer size for a relatively small amount of engineering effort, so it is generally a positive thing, and despite it being a major version bump, should be something most implementations can pick-up pretty quickly. Cheers, Micah On Wed, Jun 24, 2026 at 1:47 PM Daniel Weeks <[email protected]> wrote: > I think the main issue is that we should be coordinating and aligning on > what we want to include in a major semver update and not just forcing a new > version because we made an incompatible change. > > I'm strongly in favor of SemVer, but how we go about implementing it > matters. > > Most of the in progress proposals (modular footer, non-contiguous pages, > encodings, types) don't necessitate a major version update, so moving > forward with this change just to cause a bump seems like the wrong way to > get started on more coordinated releases. > > I would like to hear more input from others on the proposed updates to the > magic number/footer. > > -Dan > > On Wed, Jun 24, 2026 at 12:50 PM Ed Seidl <[email protected]> wrote: > > > I'm largely in agreement with Micah, but want to address a few points. > > > > I proposed the path_in_schema changes in part to test our documented > > process for dealing with forward incompatible changes. This process was > > found to be wanting, so it's good we're addressing its deficiencies. I do > > think that change (and other structural changes) should wait until we > have > > a clear mechanism in place to signal to old readers that these aren't the > > bits you're looking for, and I feel changing the file magic is pretty > much > > the > > only way to do this. > > > > I also agree with Micah that an empty list for path_in_schema is worse > than > > omitting the field (although using path_in_schema in the way Alkis > > described > > has its own issues, chief among them the fact that Parquet does not > dictate > > that column names are unique!?!). > > > > I agree that adopting SemVer and a mechanism to provide some > informational > > version string to users can be handled separately. > > > > Thanks, > > Ed > > > > On 2026/06/24 17:50:57 Micah Kornfield wrote: > > > Hi Dan, > > > > > > > I took a look at the proposed changes in #588 > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > think > > > > that's the right way forward (at least not entirely). > > > > > > > > > Thank you for the review. I responded below with comments inline but > > while > > > I think I understand some of your reservations, I'm not actually sure > > what > > > you believe the right path forward is. Could you elaborate on what you > > > believe the right path forward is? Specifically, where do we want to > get > > > to close this conversation? > > > > > > For path_in_schema, I think we can leverage the current precedent of > > having > > > > a configurable flag to omit and produce an empty list. > > > > > > > > > I don't think we can emit an empty list here, it needs to be absent to > > > prevent data corruption (so thrift parsers fail hard). Per Alkis's > email > > > on the the other thread [1] some readers are using this as a column > > > resolution path, it's not clear how defensively they are coded and > > omitting > > > the list could cause data corruption (reading back all null values for > > > columns that don't populate the field). We need something that breaks > > hard > > > for readers. I don't want to get into a state where ambiguously > > > interpretable parquet files exist. > > > > > > > > > > but we > > > > wouldn't default or change this to optional until there's a formal V3 > > > > release. > > > > > > > > > I think this might be one of the points of different philosophies. I > > think > > > we should be releasing off of head and once a forwards incompatible > > change > > > is merged, the next release automatically gets a version bump. But it > > > sounds like you might have different ideas here? > > > > > > I agree with adopting semver for parquet format releases > > > > > > > > > Great, hopefully everyone else is ok with this. > > > > > > > > > > (though there's > > > > still the question around how that aligns with the parquet footer > > version). > > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > capabilities bitset, but I don't think that's something we need to > > commit > > > > to just yet, so I don't feel there's a need to rush to a decision on > > that > > > > point. > > > > > > > > > For all intents and purposes the version field in the parquet footer > > > version is useless today. There are several mainstream readers that > > don't > > > take any action on the version number, and there is at least one writer > > > that allows it to be configurable or hard-codes it. Introducing a new > > > magic number provides a mechanism to reset this. However, I think the > > > version number should be treated as informational only (i.e. it > shouldn't > > > be validated by readers) and having a fine grained enumeration of > > features > > > a reader needs to understand is a better approach for the following > > reasons: > > > > > > 1. It allows for readers to produce finer grained error messages for > the > > > options it doesn't support (for the 1 time cost of changing the magic > > > number, which will produce "not a parquet file") > > > 2. To maximize compatibility a writer only needs to record which > forward > > > incompatible features it used, it doesn't need to recompute the lowest > > > possible major version the features fell into. For example, suppose a > > > reader only knows how to read version 3. If new a writer chooses to > > write > > > version "4" but doesn't actually use a forward incompatible changes > (e.g. > > > Version 4 was created because of a new encoding scheme that didn't > happen > > > to be used), the writing code could still write "4" as that is what was > > > requested but the reader could still read the produced file since there > > are > > > no forward incompatible bits set. > > > > > > Thanks, > > > Micah > > > > > > > > > [1] https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j > > > > > > > > > > > > > > > > > > On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> > wrote: > > > > > > > Micah, > > > > > > > > I took a look at the proposed changes in #588 > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > think > > > > that's the right way forward (at least not entirely). > > > > > > > > What I see is three independent changes that we shouldn't lump all > > > > together: > > > > > > > > 1. Parquet Format Semver > > > > 2. Addressing path_in_schema > > > > 3. Introducing a new magic number (PARX) > > > > > > > > I agree with adopting semver for parquet format releases (though > > there's > > > > still the question around how that aligns with the parquet footer > > version). > > > > > > > > For path_in_schema, I think we can leverage the current precedent of > > having > > > > a configurable flag to omit and produce an empty list. This would > > address > > > > the space concerns for those who want to opt out of compatibility, > but > > we > > > > wouldn't default or change this to optional until there's a formal V3 > > > > release. > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > capabilities bitset, but I don't think that's something we need to > > commit > > > > to just yet, so I don't feel there's a need to rush to a decision on > > that > > > > point. > > > > > > > > -Dan > > > > > > > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield < > > [email protected]> > > > > wrote: > > > > > > > > > To try to move the conversation forward I made two PRs: > > > > > > > > > > 1. parquet-format ( > https://github.com/apache/parquet-format/pull/588 > > ): > > > > > TL;DR; > > > > > - Changes language to use recommended specification > version > > as > > > > the > > > > > mechanism for configuration. > > > > > - Commits to using SemVer for parquet-format releases > going > > > > > forward (all forwards incompatible changes, including encodings, > etc) > > > > will > > > > > bump the version number. > > > > > - Adds a proposal for a new PARX magic number that has a > new > > > > fixed > > > > > length component to the file footer composed of (metadata_len, > > feature > > > > > bitmap, CRC for footer and 'PARX' trailer). This also unifies > > encrypted > > > > and > > > > > unencrypted parquet files. > > > > > > > > > > > > > > > 2. A POC in Rust on how this could be implemented ( > > > > > https://github.com/apache/arrow-rs/pull/10177), including its > usage > > with > > > > > path_in_footer > > > > > > > > > > I think this meets the implicit requirements discussed in this > > thread. > > > > > Namely: > > > > > > > > > > 1. It allows users to think about versions in a canonical way for > > > > feature > > > > > enablement. > > > > > 2. Keep recommendations that won't push default versions too > > quickly. > > > > > 3. Allows for continuous and iterative releases of the > > specification. > > > > > 4. Allows flexibility for readers to determine at a granular level > > if > > > > they > > > > > can properly read the file (and forces a single forward > incompatible > > > > change > > > > > so we aren't relying on guesswork for when a writer feature can be > > safely > > > > > enabled). > > > > > 5. Limits the need for new magic numbers past a single new value. > > > > > > > > > > Please let me know if I missed something. If we can gain consensus > > > > around > > > > > this, I can add a Java implementation so we can adopt the changes > and > > > > vote > > > > > on them. > > > > > > > > > > Cheers, > > > > > Micah > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > I agree this is getting a bit too complicated, I feel like > everyone > > > > here > > > > > > understands versions as does the wider community. Why not just > > start > > > > > there > > > > > > and add other techniques if that fails to work properly or be > > effective > > > > > for > > > > > > communication. > > > > > > > > > > > > > > > > > > I think we are better of just choosing something simple and going > > > > forward > > > > > > rather than deliberating, the worst thing that happens is that we > > have > > > > to > > > > > > make a different choice later. I’m not sure that’s worse than > > sitting > > > > on > > > > > > what we currently have and not being able to make progress on new > > > > > encodings > > > > > > or footers. > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield < > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Thank you for the feedback Andrew. Practically speaking, I > > wonder if > > > > we > > > > > > > should have two separate notions of feature bundling: > > > > > > > > > > > > > > 1. Specification version (this would be primary and risks > using > > > > > features > > > > > > > that aren't widely adopted). > > > > > > > 2. Presets - Gives users a different way of configuring things > > that > > > > > > allows > > > > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > This discussion and the proposals seem to have gotten very > > > > > complicated. > > > > > > > > People not on this mailing list or not doing regular > > development on > > > > > > > Parquet > > > > > > > > would probably benefit from simplicity. Most people are used > to > > > > > version > > > > > > > > numbers without worrying about various types of compatibility > > -- > > > > > > readers > > > > > > > > can simply state "I can read version 3-7", for example. Users > > > > > > understand > > > > > > > > this. People writing files can also easily understand "I want > > to > > > > > write > > > > > > a > > > > > > > > version 6 file because version 6 supports feature X that I > > want." > > > > or > > > > > > "I > > > > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > > > > > > > I don't really care about the details of a solution, but > please > > > > keep > > > > > in > > > > > > > > mind that a more simple solution probably increases > > accessibility > > > > for > > > > > > the > > > > > > > > widest range of people. > > > > > > > > > > > > > > > > -- > > > > > > > > Andrew Bell > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
