> > Sorry, I didn't think that was one that was valid and under consideration. > Having two grouping systems is likely overly complicated for users and > administrators to reason about, and I thought that we were aligned in that > we need something easily understood and well-defined.
Sorry to clarify, I don't think we need to consider it for now. It is something that can maybe come in the future if there is a need. On Thu, Jun 25, 2026 at 7:09 PM Ryan Blue <[email protected]> wrote: > > I think I mentioned it above, but I see presets as something that can > augment a numeric version > > Sorry, I didn't think that was one that was valid and under consideration. > Having two grouping systems is likely overly complicated for users and > administrators to reason about, and I thought that we were aligned in that > we need something easily understood and well-defined. > > Not only that, we have been talking about this grouping as defining > compatibility. You can't do that with two disjoint groupings -- you can't > fail to read a version that is higher than what is supported, but also fail > to read a preset that isn't supported. At best, you could make one a subset > of the other, so, for example, a preset is a subset of features in a > version. That doesn't seem to accomplish a lot. Why would you not just have > more frequent versions? > > A mashup of schemes doesn't seem like the right direction to me, but maybe > there's something with the other schemes that you think is missing? I'm > reasonably confident that the version scheme is complete, given that we've > been using it for most of a decade -- and we inherited the ideas from > Parquet to begin with. > > If we aren't solving for something missing, then I think we should focus on > decision points that bring us closer to consensus. The main one is how we > want to decide what goes into these groups. I think we're closer than we > all realize to consensus if we strip away some of these distractions. > > Ryan > > On Thu, Jun 25, 2026 at 1:15 PM Micah Kornfield <[email protected]> > wrote: > > > > > > > The only alternative I see is the preset scheme, > > > which has some traction from Micah and Antoine. > > > > > > I think I mentioned it above, but I see presets as something that can > > augment a numeric version. It is likely that not all implementations > will > > support all features whatever versioning scheme is used. Having an end > user > > facing feature that can let them select a common denominator across > systems > > seems useful, but I don't think it should be the primary selection > > mechanism (nor something that necessarily needs to be standardized). > > > > Cheers, > > Micah > > > > On Thu, Jun 25, 2026 at 12:35 PM Ryan Blue <[email protected]> wrote: > > > > > Since I've been out, I'm coming back to this thread and trying to make > > > sense of it. I think that's helpful because I'm trying to separate the > > > various points of discussion. Here's a quick summary of where I think > we > > > are: > > > > > > * We are primarily talking about forward-incompatible changes > > > * Parquet needs to group or "bundle" those changes together; I think > > > Alkis's summary was good and haven't seen main objections to its main > > > points (just to implementation details) > > > * Our system should be simple and understandable (+1 to Andrew's (Bell) > > and > > > Russell's comments) > > > > > > I think we have a few things left to decide for the system that we > > choose: > > > > > > 1. How to identify groups > > > 2. How to include changes in groups > > > 3. Where to track the targeted compatibility that will work for older > > > readers and similar technical details > > > > > > I propose that we focus on the first two decisions before we talk in > much > > > detail about the third or open parquet-format PRs. Unless we understand > > > high level "how it works", we can't really plan the implementation > > changes. > > > I think that staying focused on the process we are building and > > sequencing > > > these decisions will help us make faster progress. I'd also like to ask > > > that we start more focused threads for other pieces of this discussion, > > > rather than bringing tangents here like debating the details of > whether a > > > specific change is forward-compatible or could be made > > forward-compatible. > > > > > > The rest of this email is concerned with those two decision points. > > > > > > > I think the core tension in this thread is between 2 choices: > Versions, > > > ... Bundles/Presets/etc. > > > > > > In my mind, these aren't actually that different, or at least versions > > and > > > presets are both examples of bundles or groups. The distinction between > > > versions and presets are (1) versions use numeric increasing > identifiers, > > > and (2) features are included in the next version until it is closed > and > > > adopted by a vote (vs. an automatic process, like what is in 3 > > > implementations by a certain point in time). These two differences are > > > basically the same thing as the first two decisions I outlined above. > > > > > > I think that we have support for numeric versions or identifiers from > > > several people. SemVer is a popular suggestion and I think Russell, > Dan, > > > Andrew (Lamb), and I would support using the Iceberg numbered version > > model > > > or similar. I recommend that we move forward assuming we will use a > > numeric > > > scheme similar to SemVer (discuss in the other thread?) but we can > > revisit > > > this after we decide how to include features. If we were to choose an > > > automatic/mechanistic process then the preset identifier scheme makes a > > lot > > > more sense! > > > > > > As for how to include features in our bundles / groups, I think there > is > > a > > > fair amount of support from Andrew Lamb, Dan, Russell, Andrew Bell (I > > > think), and myself for putting these changes into the next version and > > > voting when to release it. The only alternative I see is the preset > > scheme, > > > which has some traction from Micah and Antoine. > > > > > > If I'm right, then I think we're pretty close to a resolution and that > we > > > need to decide between a mechanistic preset process and a process of > > > releasing a new version by a vote. Does anyone have another idea that > is > > > significantly different? > > > > > > Ryan > > > > > > On Thu, Jun 25, 2026 at 11:07 AM Andrew Lamb <[email protected]> > > > wrote: > > > > > > > I think adopting the Iceberg model for Parquet would be a good idea, > > > > personally. > > > > > > > > On Thu, Jun 25, 2026 at 1:30 PM Ryan Blue <[email protected]> wrote: > > > > > > > > > I was out last week so I'm catching up on this thread > incrementally. > > > I'm > > > > > going to reply in shorter emails so that I'm more direct and we > don't > > > > have > > > > > 12-page emails with bullets and sub-sections and cross references > > that > > > > are > > > > > hard to read. > > > > > > > > > > I want to clarify what Iceberg does. It has, indeed, used format > > > bundles > > > > > successfully since the beginning, using version numbers that people > > can > > > > > easily understand (sounds like that would work for Andrew). Iceberg > > > > bundles > > > > > all forward-incompatible changes for release, doing at a high level > > > > exactly > > > > > what Alkis suggested. > > > > > > > > > > But there are also ways to get features early. We have introduced > > flags > > > > in > > > > > the past for people who want to use a specific feature in an > earlier > > > > > version and understand the compatibility risks. It's like importing > > > from > > > > > future in Python. > > > > > > > > > > For Parquet, this would be like adding a flag to enable ALP > encoding, > > > but > > > > > not the rest of the bundle in which support for ALP is delivered. > > This > > > > > works well for the high-level features that some people want to > > enable > > > > > early. It fits with the writer-side feature flags, but still allows > > us > > > to > > > > > bundle features together. Instead of encodings being outside of > > feature > > > > > bundles, we would consider support required in Parquet 3 and > optional > > > in > > > > > Parquet 1/2, to be enabled by a feature flag. > > > > > > > > > > Iceberg also delivers features that are forward-compatible at any > > time > > > > and > > > > > we make changes so that features are forward-compatible. For > > instance, > > > > new > > > > > metadata that older readers can safely ignore (does not cause > > > correctness > > > > > problems) is fine to add at any time. We have also added rules to > new > > > > > versions to make this possible: Iceberg v3 requires that readers > > ignore > > > > > partition functions that they do not understand, and that writers > > will > > > > not > > > > > write if using the partition function is required. > > > > > > > > > > On Fri, Jun 12, 2026 at 1:14 PM Alkis Evlogimenos via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > > This is the hybrid design Delta and Iceberg are converging on > > from > > > > > > opposite directions btw: Delta moved from monolithic protocol > > > versions > > > > to > > > > > > feature flags plus named bundles, Iceberg is discussing > decoupling > > > > > features > > > > > > from its voted versions. > > > > > > > > > > > > Micah pointed to me that this is not true for Iceberg. I failed > to > > > > double > > > > > > check my AI researcher. Iceberg is a format where bundles have > > worked > > > > > > successfully since the beginning. The feature flags discussed for > > > > Iceberg > > > > > > are on the REST endpoint not the format itself. > > > > > > > > > > > > On Fri, Jun 12, 2026 at 2:41 AM Alkis Evlogimenos < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Great discussion. It looks like we are converging on the > > important > > > > > part: > > > > > > > features need to be bundled. In the threads we used > > > > > > version/preset/epoch; I > > > > > > > will call it a bundle from here on. A bundle is a frozen set of > > > > > > features. A > > > > > > > bundle draws a clear line where we have: > > > > > > > > > > > > > > 1. the ability to co-land features coherently > > > > > > > 2. a clear path to deprecate features > > > > > > > 3. a clean UX for writes (target=bundle-2026/v3/whatever) > > > > > > > > > > > > > > To make this work we need to ratify that writers declare the > > bundle > > > > > when > > > > > > a > > > > > > > file uses its features, and readers fail the read if they don't > > > > support > > > > > > it. > > > > > > > This is the hybrid design Delta and Iceberg are converging on > > from > > > > > > opposite > > > > > > > directions btw: Delta moved from monolithic protocol versions > to > > > > > feature > > > > > > > flags plus named bundles, Iceberg is discussing decoupling > > features > > > > > from > > > > > > > its voted versions. > > > > > > > > > > > > > > Encodings already work like feature flags: a reader that can't > > > decode > > > > > an > > > > > > > encoding fails with a clear, local error. They have a safe path > > > > forward > > > > > > > today, so they can stay out of band. The tradeoff is that > > "supports > > > > > > bundle > > > > > > > X" won't cover encodings. > > > > > > > > > > > > > > The part we need to solve is structural changes: deprecating > > > > > > > path_in_schema, non-contiguous pages, a new footer. These have > no > > > > clean > > > > > > > failure mode in deployed readers. The version in the footer is > > > > > > > not salvageable as a signal. But it is a required field and we > > can > > > > use > > > > > it > > > > > > > to poison old readers! I propose: > > > > > > > > > > > > > > 1. Add a bundle field to FileMetaData (optional in thrift for > > > > > > > compatibility with existing files, mandatory to write when the > > file > > > > > uses > > > > > > > bundle features). > > > > > > > 2. Mark FileMetaData.version optional in thrift. A writer that > > sets > > > > the > > > > > > > bundle field omits version. A file carries exactly one of the > > two. > > > > > > > 3. Readers that see the bundle field must support the declared > > > bundle > > > > > or > > > > > > > fail with an error naming it. > > > > > > > > > > > > > > The trick is (2): the deployed readers I checked hard-fail at > > > footer > > > > > > parse > > > > > > > when FileMetaData.version is missing: parquet-java, arrow-cpp, > > > > > parquet-rs > > > > > > > and DuckDB. They all enforce its presence even though the spec > > says > > > > to > > > > > > > ignore its value. Old readers fail immediately on open instead > of > > > > > > tripping > > > > > > > on obscure errors later, or worse, reading bad data. > > > > > > > > > > > > > > This is a one time cost and we should pay it as early as > > possible. > > > > > Ratify > > > > > > > the first bundle's contents: path_in_schema deprecation plus > > other > > > > > small > > > > > > > cleanups. From that point on bundle aware readers fail cleanly > on > > > > > future > > > > > > > bundles. > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 8:28 PM Ryan Blue <[email protected]> > > wrote: > > > > > > > > > > > > > >> Sorry for the duplicate, but Micah said the quote blocks > didn't > > go > > > > > > through > > > > > > >> so I'm re-sending with `>` so that this is more readable. > > > > > > >> > > > > > > >> This is quite a large thread, so hopefully I am not missing > any > > > big > > > > > > points > > > > > > >> that have already been settled. > > > > > > >> > > > > > > >> If I understand correctly, it looks like there are some good > > > things > > > > > that > > > > > > >> we > > > > > > >> agree on. The most consequential is that we want to bundle > > > features > > > > > > >> together. > > > > > > >> > > > > > > >> From Antoine’s response to my email about presets, I came away > > > with > > > > > the > > > > > > >> clarification that a preset acts as a bundle of features much > > > like a > > > > > > >> version. This is a big step forward because it would be such a > > > > > challenge > > > > > > >> for users to reason about every feature individually and to > > check > > > > > > support > > > > > > >> for features across implementations. The worst outcome, in my > > > > opinion, > > > > > > >> would be leaving users or administrators to deal with a wild > > west > > > of > > > > > > >> feature flags, so I’m glad to see we’re making progress! > > > > > > >> > > > > > > >> This just leaves the details of how we want to manage those > > > feature > > > > > > >> bundles. The preset option is a mechanistic approach to > > inclusion, > > > > > while > > > > > > >> the version option relies on building consensus for features > to > > > > > > include. I > > > > > > >> think this is a good question to focus on right now because I > > > think > > > > we > > > > > > >> have > > > > > > >> yet to come to a shared understanding of both options in order > > to > > > > > > discuss > > > > > > >> the trade-offs between them. > > > > > > >> > > > > > > >> With the rest of this email, I’ll try to address some > confusion > > I > > > > saw > > > > > in > > > > > > >> the thread about how versions and/or presets would work. One > > good > > > > > thing > > > > > > to > > > > > > >> note is that we’re primarily talking about > forward-incompatible > > > > > changes: > > > > > > >> changes that would cause older readers to fail and/or read > data > > > > > > >> incorrectly. > > > > > > >> > > > > > > >> > Antoine: you also don’t know what’s in a version until the > > > version > > > > > > gets > > > > > > >> decided upon > > > > > > >> > > > > > > >> One of the strengths is that you do get to know what’s in a > > > version > > > > > > ahead > > > > > > >> of time. The process is deliberate and predictable. > > > > > > >> > > > > > > >> When we add a new (forward-incompatible) feature it > > automatically > > > > goes > > > > > > >> into > > > > > > >> the next format version. I think that we would do this using a > > > vote > > > > so > > > > > > >> that > > > > > > >> everyone here gets to take part in the decision. The features > > that > > > > we > > > > > > have > > > > > > >> agreed on are documented in the format for that version and > when > > > we > > > > > want > > > > > > >> to > > > > > > >> close the version we have another vote to adopt it. Then new > > > changes > > > > > go > > > > > > >> into the next version. > > > > > > >> > > > > > > >> This procedure ensures that we agree, via community consensus, > > on > > > > what > > > > > > >> goes > > > > > > >> into a version and we accumulate a list that is predictable > for > > > > > > >> implementations to target. > > > > > > >> > > > > > > >> > Antoine: IIRC the basis for this discussion was to inform > > > Parquet > > > > > > >> writers > > > > > > >> about which features can safely be enabled. > > > > > > >> > > > > > > >> I think this understates the problem and I prefer Andrew’s > > > summary, > > > > > > which > > > > > > >> is that we need a way for readers and writers to coordinate > > about > > > > this > > > > > > >> problem. That’s why writer flags alone are untenable because > the > > > > > number > > > > > > of > > > > > > >> things to coordinate is so large. > > > > > > >> > > > > > > >> There’s also a lot more to it than “can produce bits for X”. > > This > > > > gets > > > > > > to > > > > > > >> the hypothetical posed about Parquet 2.34. There are > > > > > > forward-incompatible > > > > > > >> features that appear readable by older clients but cause them > to > > > > > produce > > > > > > >> incorrect data. For instance, I could add a field to a page > > header > > > > > that > > > > > > >> gives an offset that should be added to all values in a page, > so > > > > that > > > > > we > > > > > > >> can pack values in smaller bit widths. Older readers would > skip > > > the > > > > > > offset > > > > > > >> and produce bad values. That’s one reason why older readers > > should > > > > > fail > > > > > > >> for > > > > > > >> newer feature bundles. > > > > > > >> > > > > > > >> Clearly documenting reader responsibilities is a big part of > > this > > > > work > > > > > > as > > > > > > >> well. If we assume that we will have readers that will fail > > (which > > > > we > > > > > > >> could > > > > > > >> do), we have to design features that force them to fail. And > > then > > > we > > > > > > have > > > > > > >> to deal with bad error messages to users. So thinking through > > how > > > > our > > > > > > >> system of working with feature bundles is really important, > not > > > just > > > > > for > > > > > > >> collecting sets of writer flags. > > > > > > >> > > > > > > >> > There is no easy-to-read list of changes unless I am missing > > > > > > something. > > > > > > >> > > > > > > >> Yes, part of the problem is that the table of features was > > > removed, > > > > > > which > > > > > > >> is a big part of what caused the current confusion about what > v2 > > > is. > > > > > > >> > > > > > > >> But this is a problem for both ways to manage version bundles, > > > > right? > > > > > > >> > > > > > > >> > 2) Let’s say Parquet 2.34 introduces features A and B. Let’s > > > also > > > > > say > > > > > > a > > > > > > >> > Parquet reader implements feature A but not feature B. What > > > should > > > > > > this > > > > > > >> > reader do if you give it a file that has version 2.34 > recorded > > > in > > > > > the > > > > > > >> > metadata? Should it error out (but perhaps the file only > uses > > > > > feature > > > > > > >> > A)? Or should it not error out (but perhaps the file uses > > > feature > > > > > B)? > > > > > > >> > > > > > > >> I mentioned this above, but I think this affects both > strategies > > > for > > > > > > >> bundling features and doesn’t really distinguish between them. > > > > > > >> > > > > > > >> I will note, though, that we have a clear rule about this in > the > > > > > Iceberg > > > > > > >> community that works well for versions. Readers must fail if > > they > > > > > don’t > > > > > > >> recognize a version. If a reader knows about a version and > has a > > > > > feature > > > > > > >> gap, it can look for the missing feature and fail with a good > > > error > > > > > > >> message > > > > > > >> but otherwise proceed. The main thing is that the feature > bundle > > > > > > (version > > > > > > >> in Iceberg) is understood and is the primary way the group is > > > > > > coordinated. > > > > > > >> > > > > > > >> > Historically it’s been quite common to have this kind of > > jagged > > > > > > feature > > > > > > >> adoption where implementations do not necessarily implement > > > features > > > > > in > > > > > > >> the > > > > > > >> chronological order of their appearance in parquet-format. > > > > > > >> > > > > > > >> This is something that we are addressing with this discussion! > > The > > > > > goal > > > > > > is > > > > > > >> a way to coordinate between readers and writers, right? > > > > > > >> > > > > > > >> > Readers already error out when then encounter an unknown > > > encoding > > > > > in a > > > > > > >> column they are asked to reader. What do we gain by having > them > > > also > > > > > > check > > > > > > >> a version number? > > > > > > >> > > > > > > >> You cannot guarantee compatibility with reader failures alone, > > and > > > > you > > > > > > >> often want better support for even missing features than you > get > > > > with > > > > > > >> whatever failure occurs. I think Dan’s doc has a good section > on > > > > this. > > > > > > >> > > > > > > >> Ryan > > > > > > >> > > > > > > >> On Thu, Jun 11, 2026 at 1:52 AM Antoine Pitrou < > > > [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Le 10/06/2026 à 16:40, Micah Kornfield a écrit : > > > > > > >> > > > > > > > > >> > > In any case, this does not seem to be solving the problem > of > > > > "as a > > > > > > >> user, > > > > > > >> > >> how do I enable features safely". > > > > > > >> > > > > > > > > >> > > Can you elaborate? Every feature listed after 2023, hass > > the > > > > year > > > > > > it > > > > > > >> was > > > > > > >> > > introduced in parenthesis next to it. > > > > > > >> > > I think this in addition to > > > > > > >> > > the table showing the version that everything was > supported > > > in, > > > > > can > > > > > > >> > give a > > > > > > >> > > user a pretty good idea of what might be safe > > > > > > >> > > > > > > > >> > Ok, so concretely, what is a user supposed to do with these > > > > tables? > > > > > > >> > > > > > > > >> > I'm sorry for being so stubborn and insistent, but Parquet > > files > > > > are > > > > > > >> > produced routinely by data scientists and other people with > no > > > > > expert > > > > > > >> > knowledge of Parquet internals. > > > > > > >> > > > > > > > >> > If "how to produce an optimized Parquet file" takes an > entire > > > > > > paragraph > > > > > > >> > to explain and requires diving into tables of features, then > > we > > > > > > haven't > > > > > > >> > solved the problem. > > > > > > >> > > > > > > > >> > > > > > > > >> > (also, even I don't know what to do with the information of > > > "Arrow > > > > > C++ > > > > > > >> > does not support 2025 features": what does it bring to the > > > > reader?) > > > > > > >> > > > > > > > >> > Regards > > > > > > >> > > > > > > > >> > Antoine. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
