I think the core tension in this thread is between 2 choices: Versions: pros: easily understood, commonly used elsewhere in the industry cons: blunt scope (e.g. using **any** feature of "v3" seemingly requires implementing **all** features of "v3" which is a lot of work: variant, geo, etc)
Feature bundles / presets / packages ,etc: pros: finer scope (e.g. users can use a subset of features important to their usecase without implementing all the features) cons: harder to explain / implement / configure The technical proposals I see fall into one or the other category; I doubt we'll get consensus unless this core tension is resolved (will we do versions, or do we want something finer grained). Andrew p.s. I think we could make either work and I have no strong preference between them On Thu, Jun 25, 2026 at 4:11 AM Micah Kornfield <[email protected]> wrote: > Hi Dan, > > Responses inline. > > I think the main issue is that we should be coordinating and aligning on > > what we want to include in a major semver update and not just forcing a > new > > version because we made an incompatible change. > > > To have a practical discussion I think it would help to post a PR with a > proposal to CONTRIBUTING.md on how such a process would work so we can > evaluate the difference with the what is currently in contributing.md, the > strawman posted earlier in the thread ( > https://github.com/apache/parquet-format/pull/588) and how you are > thinking > about the problem. One of my biggest concerns is having mechanisms in place > to avoid falling into the "V2" trap. > > > > Most of the in progress proposals (modular footer, non-contiguous pages, > > encodings, types) don't necessitate a major version update, > > > Non-contiguous pages and encodings are forward-incompatible changes and in > my mind would definitely necessitate a major version bump. The modular > footer, depending on implementation, would also likely fall into a > forward-incompatible change category. I agree new types would not require a > major version upgrade, as they are already classified as > forward-compatible. Since there seems to be some misalignment here, > covering how you are thinking about this working as part of the process in > a PR proposal would be valuable to further the discussion. > > so moving > > forward with this change just to cause a bump seems like the wrong way to > > get started on more coordinated releases. > > > While Ed mentioned this was a test-case for a forward incompatible change, > I at least did not see this as the primary focus of the proposal. My > understanding is that this change offers real practical benefits for > reducing the footer size for a relatively small amount of engineering > effort, so it is generally a positive thing, and despite it being a major > version bump, should be something most implementations can pick-up pretty > quickly. > > Cheers, > Micah > > > On Wed, Jun 24, 2026 at 1:47 PM Daniel Weeks <[email protected]> wrote: > > > I think the main issue is that we should be coordinating and aligning on > > what we want to include in a major semver update and not just forcing a > new > > version because we made an incompatible change. > > > > I'm strongly in favor of SemVer, but how we go about implementing it > > matters. > > > > Most of the in progress proposals (modular footer, non-contiguous pages, > > encodings, types) don't necessitate a major version update, so moving > > forward with this change just to cause a bump seems like the wrong way to > > get started on more coordinated releases. > > > > I would like to hear more input from others on the proposed updates to > the > > magic number/footer. > > > > -Dan > > > > On Wed, Jun 24, 2026 at 12:50 PM Ed Seidl <[email protected]> wrote: > > > > > I'm largely in agreement with Micah, but want to address a few points. > > > > > > I proposed the path_in_schema changes in part to test our documented > > > process for dealing with forward incompatible changes. This process was > > > found to be wanting, so it's good we're addressing its deficiencies. I > do > > > think that change (and other structural changes) should wait until we > > have > > > a clear mechanism in place to signal to old readers that these aren't > the > > > bits you're looking for, and I feel changing the file magic is pretty > > much > > > the > > > only way to do this. > > > > > > I also agree with Micah that an empty list for path_in_schema is worse > > than > > > omitting the field (although using path_in_schema in the way Alkis > > > described > > > has its own issues, chief among them the fact that Parquet does not > > dictate > > > that column names are unique!?!). > > > > > > I agree that adopting SemVer and a mechanism to provide some > > informational > > > version string to users can be handled separately. > > > > > > Thanks, > > > Ed > > > > > > On 2026/06/24 17:50:57 Micah Kornfield wrote: > > > > Hi Dan, > > > > > > > > > I took a look at the proposed changes in #588 > > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > > think > > > > > that's the right way forward (at least not entirely). > > > > > > > > > > > > Thank you for the review. I responded below with comments inline but > > > while > > > > I think I understand some of your reservations, I'm not actually sure > > > what > > > > you believe the right path forward is. Could you elaborate on what > you > > > > believe the right path forward is? Specifically, where do we want to > > get > > > > to close this conversation? > > > > > > > > For path_in_schema, I think we can leverage the current precedent of > > > having > > > > > a configurable flag to omit and produce an empty list. > > > > > > > > > > > > I don't think we can emit an empty list here, it needs to be absent > to > > > > prevent data corruption (so thrift parsers fail hard). Per Alkis's > > email > > > > on the the other thread [1] some readers are using this as a column > > > > resolution path, it's not clear how defensively they are coded and > > > omitting > > > > the list could cause data corruption (reading back all null values > for > > > > columns that don't populate the field). We need something that > breaks > > > hard > > > > for readers. I don't want to get into a state where ambiguously > > > > interpretable parquet files exist. > > > > > > > > > > > > > but we > > > > > wouldn't default or change this to optional until there's a formal > V3 > > > > > release. > > > > > > > > > > > > I think this might be one of the points of different philosophies. I > > > think > > > > we should be releasing off of head and once a forwards incompatible > > > change > > > > is merged, the next release automatically gets a version bump. But > it > > > > sounds like you might have different ideas here? > > > > > > > > I agree with adopting semver for parquet format releases > > > > > > > > > > > > Great, hopefully everyone else is ok with this. > > > > > > > > > > > > > (though there's > > > > > still the question around how that aligns with the parquet footer > > > version). > > > > > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > > capabilities bitset, but I don't think that's something we need to > > > commit > > > > > to just yet, so I don't feel there's a need to rush to a decision > on > > > that > > > > > point. > > > > > > > > > > > > For all intents and purposes the version field in the parquet footer > > > > version is useless today. There are several mainstream readers that > > > don't > > > > take any action on the version number, and there is at least one > writer > > > > that allows it to be configurable or hard-codes it. Introducing a > new > > > > magic number provides a mechanism to reset this. However, I think > the > > > > version number should be treated as informational only (i.e. it > > shouldn't > > > > be validated by readers) and having a fine grained enumeration of > > > features > > > > a reader needs to understand is a better approach for the following > > > reasons: > > > > > > > > 1. It allows for readers to produce finer grained error messages for > > the > > > > options it doesn't support (for the 1 time cost of changing the magic > > > > number, which will produce "not a parquet file") > > > > 2. To maximize compatibility a writer only needs to record which > > forward > > > > incompatible features it used, it doesn't need to recompute the > lowest > > > > possible major version the features fell into. For example, suppose > a > > > > reader only knows how to read version 3. If new a writer chooses to > > > write > > > > version "4" but doesn't actually use a forward incompatible changes > > (e.g. > > > > Version 4 was created because of a new encoding scheme that didn't > > happen > > > > to be used), the writing code could still write "4" as that is what > was > > > > requested but the reader could still read the produced file since > there > > > are > > > > no forward incompatible bits set. > > > > > > > > Thanks, > > > > Micah > > > > > > > > > > > > [1] https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> > > wrote: > > > > > > > > > Micah, > > > > > > > > > > I took a look at the proposed changes in #588 > > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > > think > > > > > that's the right way forward (at least not entirely). > > > > > > > > > > What I see is three independent changes that we shouldn't lump all > > > > > together: > > > > > > > > > > 1. Parquet Format Semver > > > > > 2. Addressing path_in_schema > > > > > 3. Introducing a new magic number (PARX) > > > > > > > > > > I agree with adopting semver for parquet format releases (though > > > there's > > > > > still the question around how that aligns with the parquet footer > > > version). > > > > > > > > > > For path_in_schema, I think we can leverage the current precedent > of > > > having > > > > > a configurable flag to omit and produce an empty list. This would > > > address > > > > > the space concerns for those who want to opt out of compatibility, > > but > > > we > > > > > wouldn't default or change this to optional until there's a formal > V3 > > > > > release. > > > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > > capabilities bitset, but I don't think that's something we need to > > > commit > > > > > to just yet, so I don't feel there's a need to rush to a decision > on > > > that > > > > > point. > > > > > > > > > > -Dan > > > > > > > > > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > To try to move the conversation forward I made two PRs: > > > > > > > > > > > > 1. parquet-format ( > > https://github.com/apache/parquet-format/pull/588 > > > ): > > > > > > TL;DR; > > > > > > - Changes language to use recommended specification > > version > > > as > > > > > the > > > > > > mechanism for configuration. > > > > > > - Commits to using SemVer for parquet-format releases > > going > > > > > > forward (all forwards incompatible changes, including encodings, > > etc) > > > > > will > > > > > > bump the version number. > > > > > > - Adds a proposal for a new PARX magic number that has a > > new > > > > > fixed > > > > > > length component to the file footer composed of (metadata_len, > > > feature > > > > > > bitmap, CRC for footer and 'PARX' trailer). This also unifies > > > encrypted > > > > > and > > > > > > unencrypted parquet files. > > > > > > > > > > > > > > > > > > 2. A POC in Rust on how this could be implemented ( > > > > > > https://github.com/apache/arrow-rs/pull/10177), including its > > usage > > > with > > > > > > path_in_footer > > > > > > > > > > > > I think this meets the implicit requirements discussed in this > > > thread. > > > > > > Namely: > > > > > > > > > > > > 1. It allows users to think about versions in a canonical way > for > > > > > feature > > > > > > enablement. > > > > > > 2. Keep recommendations that won't push default versions too > > > quickly. > > > > > > 3. Allows for continuous and iterative releases of the > > > specification. > > > > > > 4. Allows flexibility for readers to determine at a granular > level > > > if > > > > > they > > > > > > can properly read the file (and forces a single forward > > incompatible > > > > > change > > > > > > so we aren't relying on guesswork for when a writer feature can > be > > > safely > > > > > > enabled). > > > > > > 5. Limits the need for new magic numbers past a single new > value. > > > > > > > > > > > > Please let me know if I missed something. If we can gain > consensus > > > > > around > > > > > > this, I can add a Java implementation so we can adopt the changes > > and > > > > > vote > > > > > > on them. > > > > > > > > > > > > Cheers, > > > > > > Micah > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I agree this is getting a bit too complicated, I feel like > > everyone > > > > > here > > > > > > > understands versions as does the wider community. Why not just > > > start > > > > > > there > > > > > > > and add other techniques if that fails to work properly or be > > > effective > > > > > > for > > > > > > > communication. > > > > > > > > > > > > > > > > > > > > > I think we are better of just choosing something simple and > going > > > > > forward > > > > > > > rather than deliberating, the worst thing that happens is that > we > > > have > > > > > to > > > > > > > make a different choice later. I’m not sure that’s worse than > > > sitting > > > > > on > > > > > > > what we currently have and not being able to make progress on > new > > > > > > encodings > > > > > > > or footers. > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield < > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thank you for the feedback Andrew. Practically speaking, I > > > wonder if > > > > > we > > > > > > > > should have two separate notions of feature bundling: > > > > > > > > > > > > > > > > 1. Specification version (this would be primary and risks > > using > > > > > > features > > > > > > > > that aren't widely adopted). > > > > > > > > 2. Presets - Gives users a different way of configuring > things > > > that > > > > > > > allows > > > > > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > This discussion and the proposals seem to have gotten very > > > > > > complicated. > > > > > > > > > People not on this mailing list or not doing regular > > > development on > > > > > > > > Parquet > > > > > > > > > would probably benefit from simplicity. Most people are > used > > to > > > > > > version > > > > > > > > > numbers without worrying about various types of > compatibility > > > -- > > > > > > > readers > > > > > > > > > can simply state "I can read version 3-7", for example. > Users > > > > > > > understand > > > > > > > > > this. People writing files can also easily understand "I > want > > > to > > > > > > write > > > > > > > a > > > > > > > > > version 6 file because version 6 supports feature X that I > > > want." > > > > > or > > > > > > > "I > > > > > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > > > > > > > > > I don't really care about the details of a solution, but > > please > > > > > keep > > > > > > in > > > > > > > > > mind that a more simple solution probably increases > > > accessibility > > > > > for > > > > > > > the > > > > > > > > > widest range of people. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Andrew Bell > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
