Hi Andrew, Versions:
Feature bundles / presets / packages ,etc: > The technical proposals I see fall into one or the other category; I doubt > we'll get consensus unless this core tension is resolved (will we do > versions, or do we want something finer grained). I think one issue I'm having is we've tossed around a lot of ideas on the thread, but I think there has been a lack of precision on what exactly we are proposing. I believe this could be the tension but either way having something concrete that we can evaluate makes sense. The technical proposals I see fall into one or the other category; I doubt > we'll get consensus unless this core tension is resolved (will we do > versions, or do we want something finer grained). This might be a proposal that draws ire from all sides; but why not both? 1. Version is clear, unambiguous and yes it includes everything in the spec but allows implementations on the cutting edge to move quickly without curation. 2. "Feature bundles / presets / packages ,etc:" require some level of discernment either automated or considered but this I think always has to lag the actual introduction of a feature. Otherwise it requires making prescient judgement calls on usage/importance. Thanks, Micah On Thu, Jun 25, 2026 at 3:36 AM Andrew Lamb <[email protected]> wrote: > I think the core tension in this thread is between 2 choices: > > Versions: > pros: easily understood, commonly used elsewhere in the industry > cons: blunt scope (e.g. using **any** feature of "v3" seemingly requires > implementing **all** features of "v3" which is a lot of work: variant, geo, > etc) > > Feature bundles / presets / packages ,etc: > pros: finer scope (e.g. users can use a subset of features important to > their usecase without implementing all the features) > cons: harder to explain / implement / configure > > The technical proposals I see fall into one or the other category; I doubt > we'll get consensus unless this core tension is resolved (will we do > versions, or do we want something finer grained). > > Andrew > > p.s. I think we could make either work and I have no strong preference > between them > > On Thu, Jun 25, 2026 at 4:11 AM Micah Kornfield <[email protected]> > wrote: > > > Hi Dan, > > > > Responses inline. > > > > I think the main issue is that we should be coordinating and aligning on > > > what we want to include in a major semver update and not just forcing a > > new > > > version because we made an incompatible change. > > > > > > To have a practical discussion I think it would help to post a PR with a > > proposal to CONTRIBUTING.md on how such a process would work so we can > > evaluate the difference with the what is currently in contributing.md, > the > > strawman posted earlier in the thread ( > > https://github.com/apache/parquet-format/pull/588) and how you are > > thinking > > about the problem. One of my biggest concerns is having mechanisms in > place > > to avoid falling into the "V2" trap. > > > > > > > Most of the in progress proposals (modular footer, non-contiguous > pages, > > > encodings, types) don't necessitate a major version update, > > > > > > Non-contiguous pages and encodings are forward-incompatible changes and > in > > my mind would definitely necessitate a major version bump. The modular > > footer, depending on implementation, would also likely fall into a > > forward-incompatible change category. I agree new types would not > require a > > major version upgrade, as they are already classified as > > forward-compatible. Since there seems to be some misalignment here, > > covering how you are thinking about this working as part of the process > in > > a PR proposal would be valuable to further the discussion. > > > > so moving > > > forward with this change just to cause a bump seems like the wrong way > to > > > get started on more coordinated releases. > > > > > > While Ed mentioned this was a test-case for a forward incompatible > change, > > I at least did not see this as the primary focus of the proposal. My > > understanding is that this change offers real practical benefits for > > reducing the footer size for a relatively small amount of engineering > > effort, so it is generally a positive thing, and despite it being a major > > version bump, should be something most implementations can pick-up pretty > > quickly. > > > > Cheers, > > Micah > > > > > > On Wed, Jun 24, 2026 at 1:47 PM Daniel Weeks <[email protected]> wrote: > > > > > I think the main issue is that we should be coordinating and aligning > on > > > what we want to include in a major semver update and not just forcing a > > new > > > version because we made an incompatible change. > > > > > > I'm strongly in favor of SemVer, but how we go about implementing it > > > matters. > > > > > > Most of the in progress proposals (modular footer, non-contiguous > pages, > > > encodings, types) don't necessitate a major version update, so moving > > > forward with this change just to cause a bump seems like the wrong way > to > > > get started on more coordinated releases. > > > > > > I would like to hear more input from others on the proposed updates to > > the > > > magic number/footer. > > > > > > -Dan > > > > > > On Wed, Jun 24, 2026 at 12:50 PM Ed Seidl <[email protected]> wrote: > > > > > > > I'm largely in agreement with Micah, but want to address a few > points. > > > > > > > > I proposed the path_in_schema changes in part to test our documented > > > > process for dealing with forward incompatible changes. This process > was > > > > found to be wanting, so it's good we're addressing its deficiencies. > I > > do > > > > think that change (and other structural changes) should wait until we > > > have > > > > a clear mechanism in place to signal to old readers that these aren't > > the > > > > bits you're looking for, and I feel changing the file magic is pretty > > > much > > > > the > > > > only way to do this. > > > > > > > > I also agree with Micah that an empty list for path_in_schema is > worse > > > than > > > > omitting the field (although using path_in_schema in the way Alkis > > > > described > > > > has its own issues, chief among them the fact that Parquet does not > > > dictate > > > > that column names are unique!?!). > > > > > > > > I agree that adopting SemVer and a mechanism to provide some > > > informational > > > > version string to users can be handled separately. > > > > > > > > Thanks, > > > > Ed > > > > > > > > On 2026/06/24 17:50:57 Micah Kornfield wrote: > > > > > Hi Dan, > > > > > > > > > > > I took a look at the proposed changes in #588 > > > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > > > think > > > > > > that's the right way forward (at least not entirely). > > > > > > > > > > > > > > > Thank you for the review. I responded below with comments inline > but > > > > while > > > > > I think I understand some of your reservations, I'm not actually > sure > > > > what > > > > > you believe the right path forward is. Could you elaborate on what > > you > > > > > believe the right path forward is? Specifically, where do we want > to > > > get > > > > > to close this conversation? > > > > > > > > > > For path_in_schema, I think we can leverage the current precedent > of > > > > having > > > > > > a configurable flag to omit and produce an empty list. > > > > > > > > > > > > > > > I don't think we can emit an empty list here, it needs to be absent > > to > > > > > prevent data corruption (so thrift parsers fail hard). Per Alkis's > > > email > > > > > on the the other thread [1] some readers are using this as a column > > > > > resolution path, it's not clear how defensively they are coded and > > > > omitting > > > > > the list could cause data corruption (reading back all null values > > for > > > > > columns that don't populate the field). We need something that > > breaks > > > > hard > > > > > for readers. I don't want to get into a state where ambiguously > > > > > interpretable parquet files exist. > > > > > > > > > > > > > > > > but we > > > > > > wouldn't default or change this to optional until there's a > formal > > V3 > > > > > > release. > > > > > > > > > > > > > > > I think this might be one of the points of different > philosophies. I > > > > think > > > > > we should be releasing off of head and once a forwards incompatible > > > > change > > > > > is merged, the next release automatically gets a version bump. But > > it > > > > > sounds like you might have different ideas here? > > > > > > > > > > I agree with adopting semver for parquet format releases > > > > > > > > > > > > > > > Great, hopefully everyone else is ok with this. > > > > > > > > > > > > > > > > (though there's > > > > > > still the question around how that aligns with the parquet footer > > > > version). > > > > > > > > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > > > capabilities bitset, but I don't think that's something we need > to > > > > commit > > > > > > to just yet, so I don't feel there's a need to rush to a decision > > on > > > > that > > > > > > point. > > > > > > > > > > > > > > > For all intents and purposes the version field in the parquet > footer > > > > > version is useless today. There are several mainstream readers > that > > > > don't > > > > > take any action on the version number, and there is at least one > > writer > > > > > that allows it to be configurable or hard-codes it. Introducing a > > new > > > > > magic number provides a mechanism to reset this. However, I think > > the > > > > > version number should be treated as informational only (i.e. it > > > shouldn't > > > > > be validated by readers) and having a fine grained enumeration of > > > > features > > > > > a reader needs to understand is a better approach for the following > > > > reasons: > > > > > > > > > > 1. It allows for readers to produce finer grained error messages > for > > > the > > > > > options it doesn't support (for the 1 time cost of changing the > magic > > > > > number, which will produce "not a parquet file") > > > > > 2. To maximize compatibility a writer only needs to record which > > > forward > > > > > incompatible features it used, it doesn't need to recompute the > > lowest > > > > > possible major version the features fell into. For example, > suppose > > a > > > > > reader only knows how to read version 3. If new a writer chooses > to > > > > write > > > > > version "4" but doesn't actually use a forward incompatible changes > > > (e.g. > > > > > Version 4 was created because of a new encoding scheme that didn't > > > happen > > > > > to be used), the writing code could still write "4" as that is what > > was > > > > > requested but the reader could still read the produced file since > > there > > > > are > > > > > no forward incompatible bits set. > > > > > > > > > > Thanks, > > > > > Micah > > > > > > > > > > > > > > > [1] > https://lists.apache.org/thread/8pp4v9h0hpj20kylqf8rbwl4z5xw6c7j > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 24, 2026 at 7:34 AM Daniel Weeks <[email protected]> > > > wrote: > > > > > > > > > > > Micah, > > > > > > > > > > > > I took a look at the proposed changes in #588 > > > > > > <https://github.com/apache/parquet-format/pull/588> and I don't > > > think > > > > > > that's the right way forward (at least not entirely). > > > > > > > > > > > > What I see is three independent changes that we shouldn't lump > all > > > > > > together: > > > > > > > > > > > > 1. Parquet Format Semver > > > > > > 2. Addressing path_in_schema > > > > > > 3. Introducing a new magic number (PARX) > > > > > > > > > > > > I agree with adopting semver for parquet format releases (though > > > > there's > > > > > > still the question around how that aligns with the parquet footer > > > > version). > > > > > > > > > > > > For path_in_schema, I think we can leverage the current precedent > > of > > > > having > > > > > > a configurable flag to omit and produce an empty list. This > would > > > > address > > > > > > the space concerns for those who want to opt out of > compatibility, > > > but > > > > we > > > > > > wouldn't default or change this to optional until there's a > formal > > V3 > > > > > > release. > > > > > > > > > > > > I'm not convinced about the PARX magic number and extended footer > > > > > > capabilities bitset, but I don't think that's something we need > to > > > > commit > > > > > > to just yet, so I don't feel there's a need to rush to a decision > > on > > > > that > > > > > > point. > > > > > > > > > > > > -Dan > > > > > > > > > > > > On Mon, Jun 22, 2026 at 12:04 AM Micah Kornfield < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > To try to move the conversation forward I made two PRs: > > > > > > > > > > > > > > 1. parquet-format ( > > > https://github.com/apache/parquet-format/pull/588 > > > > ): > > > > > > > TL;DR; > > > > > > > - Changes language to use recommended specification > > > version > > > > as > > > > > > the > > > > > > > mechanism for configuration. > > > > > > > - Commits to using SemVer for parquet-format releases > > > going > > > > > > > forward (all forwards incompatible changes, including > encodings, > > > etc) > > > > > > will > > > > > > > bump the version number. > > > > > > > - Adds a proposal for a new PARX magic number that > has a > > > new > > > > > > fixed > > > > > > > length component to the file footer composed of (metadata_len, > > > > feature > > > > > > > bitmap, CRC for footer and 'PARX' trailer). This also unifies > > > > encrypted > > > > > > and > > > > > > > unencrypted parquet files. > > > > > > > > > > > > > > > > > > > > > 2. A POC in Rust on how this could be implemented ( > > > > > > > https://github.com/apache/arrow-rs/pull/10177), including its > > > usage > > > > with > > > > > > > path_in_footer > > > > > > > > > > > > > > I think this meets the implicit requirements discussed in this > > > > thread. > > > > > > > Namely: > > > > > > > > > > > > > > 1. It allows users to think about versions in a canonical way > > for > > > > > > feature > > > > > > > enablement. > > > > > > > 2. Keep recommendations that won't push default versions too > > > > quickly. > > > > > > > 3. Allows for continuous and iterative releases of the > > > > specification. > > > > > > > 4. Allows flexibility for readers to determine at a granular > > level > > > > if > > > > > > they > > > > > > > can properly read the file (and forces a single forward > > > incompatible > > > > > > change > > > > > > > so we aren't relying on guesswork for when a writer feature can > > be > > > > safely > > > > > > > enabled). > > > > > > > 5. Limits the need for new magic numbers past a single new > > value. > > > > > > > > > > > > > > Please let me know if I missed something. If we can gain > > consensus > > > > > > around > > > > > > > this, I can add a Java implementation so we can adopt the > changes > > > and > > > > > > vote > > > > > > > on them. > > > > > > > > > > > > > > Cheers, > > > > > > > Micah > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I agree this is getting a bit too complicated, I feel like > > > everyone > > > > > > here > > > > > > > > understands versions as does the wider community. Why not > just > > > > start > > > > > > > there > > > > > > > > and add other techniques if that fails to work properly or be > > > > effective > > > > > > > for > > > > > > > > communication. > > > > > > > > > > > > > > > > > > > > > > > > I think we are better of just choosing something simple and > > going > > > > > > forward > > > > > > > > rather than deliberating, the worst thing that happens is > that > > we > > > > have > > > > > > to > > > > > > > > make a different choice later. I’m not sure that’s worse than > > > > sitting > > > > > > on > > > > > > > > what we currently have and not being able to make progress on > > new > > > > > > > encodings > > > > > > > > or footers. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield < > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thank you for the feedback Andrew. Practically speaking, I > > > > wonder if > > > > > > we > > > > > > > > > should have two separate notions of feature bundling: > > > > > > > > > > > > > > > > > > 1. Specification version (this would be primary and risks > > > using > > > > > > > features > > > > > > > > > that aren't widely adopted). > > > > > > > > > 2. Presets - Gives users a different way of configuring > > things > > > > that > > > > > > > > allows > > > > > > > > > for better guarantees about compatibility in the ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, June 12, 2026, Andrew Bell < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > This discussion and the proposals seem to have gotten > very > > > > > > > complicated. > > > > > > > > > > People not on this mailing list or not doing regular > > > > development on > > > > > > > > > Parquet > > > > > > > > > > would probably benefit from simplicity. Most people are > > used > > > to > > > > > > > version > > > > > > > > > > numbers without worrying about various types of > > compatibility > > > > -- > > > > > > > > readers > > > > > > > > > > can simply state "I can read version 3-7", for example. > > Users > > > > > > > > understand > > > > > > > > > > this. People writing files can also easily understand "I > > want > > > > to > > > > > > > write > > > > > > > > a > > > > > > > > > > version 6 file because version 6 supports feature X that > I > > > > want." > > > > > > or > > > > > > > > "I > > > > > > > > > > want to write version 7 because it's the latest version." > > > > > > > > > > > > > > > > > > > > I don't really care about the details of a solution, but > > > please > > > > > > keep > > > > > > > in > > > > > > > > > > mind that a more simple solution probably increases > > > > accessibility > > > > > > for > > > > > > > > the > > > > > > > > > > widest range of people. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrew Bell > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
