> > The only alternative I see is the preset scheme, > which has some traction from Micah and Antoine.
I think I mentioned it above, but I see presets as something that can augment a numeric version. It is likely that not all implementations will support all features whatever versioning scheme is used. Having an end user facing feature that can let them select a common denominator across systems seems useful, but I don't think it should be the primary selection mechanism (nor something that necessarily needs to be standardized). Cheers, Micah On Thu, Jun 25, 2026 at 12:35 PM Ryan Blue <[email protected]> wrote: > Since I've been out, I'm coming back to this thread and trying to make > sense of it. I think that's helpful because I'm trying to separate the > various points of discussion. Here's a quick summary of where I think we > are: > > * We are primarily talking about forward-incompatible changes > * Parquet needs to group or "bundle" those changes together; I think > Alkis's summary was good and haven't seen main objections to its main > points (just to implementation details) > * Our system should be simple and understandable (+1 to Andrew's (Bell) and > Russell's comments) > > I think we have a few things left to decide for the system that we choose: > > 1. How to identify groups > 2. How to include changes in groups > 3. Where to track the targeted compatibility that will work for older > readers and similar technical details > > I propose that we focus on the first two decisions before we talk in much > detail about the third or open parquet-format PRs. Unless we understand > high level "how it works", we can't really plan the implementation changes. > I think that staying focused on the process we are building and sequencing > these decisions will help us make faster progress. I'd also like to ask > that we start more focused threads for other pieces of this discussion, > rather than bringing tangents here like debating the details of whether a > specific change is forward-compatible or could be made forward-compatible. > > The rest of this email is concerned with those two decision points. > > > I think the core tension in this thread is between 2 choices: Versions, > ... Bundles/Presets/etc. > > In my mind, these aren't actually that different, or at least versions and > presets are both examples of bundles or groups. The distinction between > versions and presets are (1) versions use numeric increasing identifiers, > and (2) features are included in the next version until it is closed and > adopted by a vote (vs. an automatic process, like what is in 3 > implementations by a certain point in time). These two differences are > basically the same thing as the first two decisions I outlined above. > > I think that we have support for numeric versions or identifiers from > several people. SemVer is a popular suggestion and I think Russell, Dan, > Andrew (Lamb), and I would support using the Iceberg numbered version model > or similar. I recommend that we move forward assuming we will use a numeric > scheme similar to SemVer (discuss in the other thread?) but we can revisit > this after we decide how to include features. If we were to choose an > automatic/mechanistic process then the preset identifier scheme makes a lot > more sense! > > As for how to include features in our bundles / groups, I think there is a > fair amount of support from Andrew Lamb, Dan, Russell, Andrew Bell (I > think), and myself for putting these changes into the next version and > voting when to release it. The only alternative I see is the preset scheme, > which has some traction from Micah and Antoine. > > If I'm right, then I think we're pretty close to a resolution and that we > need to decide between a mechanistic preset process and a process of > releasing a new version by a vote. Does anyone have another idea that is > significantly different? > > Ryan > > On Thu, Jun 25, 2026 at 11:07 AM Andrew Lamb <[email protected]> > wrote: > > > I think adopting the Iceberg model for Parquet would be a good idea, > > personally. > > > > On Thu, Jun 25, 2026 at 1:30 PM Ryan Blue <[email protected]> wrote: > > > > > I was out last week so I'm catching up on this thread incrementally. > I'm > > > going to reply in shorter emails so that I'm more direct and we don't > > have > > > 12-page emails with bullets and sub-sections and cross references that > > are > > > hard to read. > > > > > > I want to clarify what Iceberg does. It has, indeed, used format > bundles > > > successfully since the beginning, using version numbers that people can > > > easily understand (sounds like that would work for Andrew). Iceberg > > bundles > > > all forward-incompatible changes for release, doing at a high level > > exactly > > > what Alkis suggested. > > > > > > But there are also ways to get features early. We have introduced flags > > in > > > the past for people who want to use a specific feature in an earlier > > > version and understand the compatibility risks. It's like importing > from > > > future in Python. > > > > > > For Parquet, this would be like adding a flag to enable ALP encoding, > but > > > not the rest of the bundle in which support for ALP is delivered. This > > > works well for the high-level features that some people want to enable > > > early. It fits with the writer-side feature flags, but still allows us > to > > > bundle features together. Instead of encodings being outside of feature > > > bundles, we would consider support required in Parquet 3 and optional > in > > > Parquet 1/2, to be enabled by a feature flag. > > > > > > Iceberg also delivers features that are forward-compatible at any time > > and > > > we make changes so that features are forward-compatible. For instance, > > new > > > metadata that older readers can safely ignore (does not cause > correctness > > > problems) is fine to add at any time. We have also added rules to new > > > versions to make this possible: Iceberg v3 requires that readers ignore > > > partition functions that they do not understand, and that writers will > > not > > > write if using the partition function is required. > > > > > > On Fri, Jun 12, 2026 at 1:14 PM Alkis Evlogimenos via dev < > > > [email protected]> wrote: > > > > > > > > This is the hybrid design Delta and Iceberg are converging on from > > > > opposite directions btw: Delta moved from monolithic protocol > versions > > to > > > > feature flags plus named bundles, Iceberg is discussing decoupling > > > features > > > > from its voted versions. > > > > > > > > Micah pointed to me that this is not true for Iceberg. I failed to > > double > > > > check my AI researcher. Iceberg is a format where bundles have worked > > > > successfully since the beginning. The feature flags discussed for > > Iceberg > > > > are on the REST endpoint not the format itself. > > > > > > > > On Fri, Jun 12, 2026 at 2:41 AM Alkis Evlogimenos < > > > > [email protected]> wrote: > > > > > > > > > Great discussion. It looks like we are converging on the important > > > part: > > > > > features need to be bundled. In the threads we used > > > > version/preset/epoch; I > > > > > will call it a bundle from here on. A bundle is a frozen set of > > > > features. A > > > > > bundle draws a clear line where we have: > > > > > > > > > > 1. the ability to co-land features coherently > > > > > 2. a clear path to deprecate features > > > > > 3. a clean UX for writes (target=bundle-2026/v3/whatever) > > > > > > > > > > To make this work we need to ratify that writers declare the bundle > > > when > > > > a > > > > > file uses its features, and readers fail the read if they don't > > support > > > > it. > > > > > This is the hybrid design Delta and Iceberg are converging on from > > > > opposite > > > > > directions btw: Delta moved from monolithic protocol versions to > > > feature > > > > > flags plus named bundles, Iceberg is discussing decoupling features > > > from > > > > > its voted versions. > > > > > > > > > > Encodings already work like feature flags: a reader that can't > decode > > > an > > > > > encoding fails with a clear, local error. They have a safe path > > forward > > > > > today, so they can stay out of band. The tradeoff is that "supports > > > > bundle > > > > > X" won't cover encodings. > > > > > > > > > > The part we need to solve is structural changes: deprecating > > > > > path_in_schema, non-contiguous pages, a new footer. These have no > > clean > > > > > failure mode in deployed readers. The version in the footer is > > > > > not salvageable as a signal. But it is a required field and we can > > use > > > it > > > > > to poison old readers! I propose: > > > > > > > > > > 1. Add a bundle field to FileMetaData (optional in thrift for > > > > > compatibility with existing files, mandatory to write when the file > > > uses > > > > > bundle features). > > > > > 2. Mark FileMetaData.version optional in thrift. A writer that sets > > the > > > > > bundle field omits version. A file carries exactly one of the two. > > > > > 3. Readers that see the bundle field must support the declared > bundle > > > or > > > > > fail with an error naming it. > > > > > > > > > > The trick is (2): the deployed readers I checked hard-fail at > footer > > > > parse > > > > > when FileMetaData.version is missing: parquet-java, arrow-cpp, > > > parquet-rs > > > > > and DuckDB. They all enforce its presence even though the spec says > > to > > > > > ignore its value. Old readers fail immediately on open instead of > > > > tripping > > > > > on obscure errors later, or worse, reading bad data. > > > > > > > > > > This is a one time cost and we should pay it as early as possible. > > > Ratify > > > > > the first bundle's contents: path_in_schema deprecation plus other > > > small > > > > > cleanups. From that point on bundle aware readers fail cleanly on > > > future > > > > > bundles. > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 8:28 PM Ryan Blue <[email protected]> wrote: > > > > > > > > > >> Sorry for the duplicate, but Micah said the quote blocks didn't go > > > > through > > > > >> so I'm re-sending with `>` so that this is more readable. > > > > >> > > > > >> This is quite a large thread, so hopefully I am not missing any > big > > > > points > > > > >> that have already been settled. > > > > >> > > > > >> If I understand correctly, it looks like there are some good > things > > > that > > > > >> we > > > > >> agree on. The most consequential is that we want to bundle > features > > > > >> together. > > > > >> > > > > >> From Antoine’s response to my email about presets, I came away > with > > > the > > > > >> clarification that a preset acts as a bundle of features much > like a > > > > >> version. This is a big step forward because it would be such a > > > challenge > > > > >> for users to reason about every feature individually and to check > > > > support > > > > >> for features across implementations. The worst outcome, in my > > opinion, > > > > >> would be leaving users or administrators to deal with a wild west > of > > > > >> feature flags, so I’m glad to see we’re making progress! > > > > >> > > > > >> This just leaves the details of how we want to manage those > feature > > > > >> bundles. The preset option is a mechanistic approach to inclusion, > > > while > > > > >> the version option relies on building consensus for features to > > > > include. I > > > > >> think this is a good question to focus on right now because I > think > > we > > > > >> have > > > > >> yet to come to a shared understanding of both options in order to > > > > discuss > > > > >> the trade-offs between them. > > > > >> > > > > >> With the rest of this email, I’ll try to address some confusion I > > saw > > > in > > > > >> the thread about how versions and/or presets would work. One good > > > thing > > > > to > > > > >> note is that we’re primarily talking about forward-incompatible > > > changes: > > > > >> changes that would cause older readers to fail and/or read data > > > > >> incorrectly. > > > > >> > > > > >> > Antoine: you also don’t know what’s in a version until the > version > > > > gets > > > > >> decided upon > > > > >> > > > > >> One of the strengths is that you do get to know what’s in a > version > > > > ahead > > > > >> of time. The process is deliberate and predictable. > > > > >> > > > > >> When we add a new (forward-incompatible) feature it automatically > > goes > > > > >> into > > > > >> the next format version. I think that we would do this using a > vote > > so > > > > >> that > > > > >> everyone here gets to take part in the decision. The features that > > we > > > > have > > > > >> agreed on are documented in the format for that version and when > we > > > want > > > > >> to > > > > >> close the version we have another vote to adopt it. Then new > changes > > > go > > > > >> into the next version. > > > > >> > > > > >> This procedure ensures that we agree, via community consensus, on > > what > > > > >> goes > > > > >> into a version and we accumulate a list that is predictable for > > > > >> implementations to target. > > > > >> > > > > >> > Antoine: IIRC the basis for this discussion was to inform > Parquet > > > > >> writers > > > > >> about which features can safely be enabled. > > > > >> > > > > >> I think this understates the problem and I prefer Andrew’s > summary, > > > > which > > > > >> is that we need a way for readers and writers to coordinate about > > this > > > > >> problem. That’s why writer flags alone are untenable because the > > > number > > > > of > > > > >> things to coordinate is so large. > > > > >> > > > > >> There’s also a lot more to it than “can produce bits for X”. This > > gets > > > > to > > > > >> the hypothetical posed about Parquet 2.34. There are > > > > forward-incompatible > > > > >> features that appear readable by older clients but cause them to > > > produce > > > > >> incorrect data. For instance, I could add a field to a page header > > > that > > > > >> gives an offset that should be added to all values in a page, so > > that > > > we > > > > >> can pack values in smaller bit widths. Older readers would skip > the > > > > offset > > > > >> and produce bad values. That’s one reason why older readers should > > > fail > > > > >> for > > > > >> newer feature bundles. > > > > >> > > > > >> Clearly documenting reader responsibilities is a big part of this > > work > > > > as > > > > >> well. If we assume that we will have readers that will fail (which > > we > > > > >> could > > > > >> do), we have to design features that force them to fail. And then > we > > > > have > > > > >> to deal with bad error messages to users. So thinking through how > > our > > > > >> system of working with feature bundles is really important, not > just > > > for > > > > >> collecting sets of writer flags. > > > > >> > > > > >> > There is no easy-to-read list of changes unless I am missing > > > > something. > > > > >> > > > > >> Yes, part of the problem is that the table of features was > removed, > > > > which > > > > >> is a big part of what caused the current confusion about what v2 > is. > > > > >> > > > > >> But this is a problem for both ways to manage version bundles, > > right? > > > > >> > > > > >> > 2) Let’s say Parquet 2.34 introduces features A and B. Let’s > also > > > say > > > > a > > > > >> > Parquet reader implements feature A but not feature B. What > should > > > > this > > > > >> > reader do if you give it a file that has version 2.34 recorded > in > > > the > > > > >> > metadata? Should it error out (but perhaps the file only uses > > > feature > > > > >> > A)? Or should it not error out (but perhaps the file uses > feature > > > B)? > > > > >> > > > > >> I mentioned this above, but I think this affects both strategies > for > > > > >> bundling features and doesn’t really distinguish between them. > > > > >> > > > > >> I will note, though, that we have a clear rule about this in the > > > Iceberg > > > > >> community that works well for versions. Readers must fail if they > > > don’t > > > > >> recognize a version. If a reader knows about a version and has a > > > feature > > > > >> gap, it can look for the missing feature and fail with a good > error > > > > >> message > > > > >> but otherwise proceed. The main thing is that the feature bundle > > > > (version > > > > >> in Iceberg) is understood and is the primary way the group is > > > > coordinated. > > > > >> > > > > >> > Historically it’s been quite common to have this kind of jagged > > > > feature > > > > >> adoption where implementations do not necessarily implement > features > > > in > > > > >> the > > > > >> chronological order of their appearance in parquet-format. > > > > >> > > > > >> This is something that we are addressing with this discussion! The > > > goal > > > > is > > > > >> a way to coordinate between readers and writers, right? > > > > >> > > > > >> > Readers already error out when then encounter an unknown > encoding > > > in a > > > > >> column they are asked to reader. What do we gain by having them > also > > > > check > > > > >> a version number? > > > > >> > > > > >> You cannot guarantee compatibility with reader failures alone, and > > you > > > > >> often want better support for even missing features than you get > > with > > > > >> whatever failure occurs. I think Dan’s doc has a good section on > > this. > > > > >> > > > > >> Ryan > > > > >> > > > > >> On Thu, Jun 11, 2026 at 1:52 AM Antoine Pitrou < > [email protected]> > > > > >> wrote: > > > > >> > > > > >> > Le 10/06/2026 à 16:40, Micah Kornfield a écrit : > > > > >> > > > > > > >> > > In any case, this does not seem to be solving the problem of > > "as a > > > > >> user, > > > > >> > >> how do I enable features safely". > > > > >> > > > > > > >> > > Can you elaborate? Every feature listed after 2023, hass the > > year > > > > it > > > > >> was > > > > >> > > introduced in parenthesis next to it. > > > > >> > > I think this in addition to > > > > >> > > the table showing the version that everything was supported > in, > > > can > > > > >> > give a > > > > >> > > user a pretty good idea of what might be safe > > > > >> > > > > > >> > Ok, so concretely, what is a user supposed to do with these > > tables? > > > > >> > > > > > >> > I'm sorry for being so stubborn and insistent, but Parquet files > > are > > > > >> > produced routinely by data scientists and other people with no > > > expert > > > > >> > knowledge of Parquet internals. > > > > >> > > > > > >> > If "how to produce an optimized Parquet file" takes an entire > > > > paragraph > > > > >> > to explain and requires diving into tables of features, then we > > > > haven't > > > > >> > solved the problem. > > > > >> > > > > > >> > > > > > >> > (also, even I don't know what to do with the information of > "Arrow > > > C++ > > > > >> > does not support 2025 features": what does it bring to the > > reader?) > > > > >> > > > > > >> > Regards > > > > >> > > > > > >> > Antoine. > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
