> Not quite, because as pointed in another thread, a major version change > in SemVer points to *backwards* incompatible changes, not forwards > incompatible changes.
I would not confuse what is being proposed with SemVer. SemVer is for API compatibility and it doesn't directly translate to formats. Saying that the spec or format follows SemVer would be a very different idea. What does cleanly translate is the idea of signaling breaking changes with a "major" version number, which is the core idea that the vote is trying to establish. (See below) > In other communities, a new major version of a file format can mean that > it's backwards incompatible. IIUC, a Zarr v3-only reader wouldn't be > able to read Zarr v2 datasets, because the metadata encoding has changed. > (see https://zarr-specs.readthedocs.io/en/latest/specs.html) This is a great example! Zarr v3 is not backward-compatible with v2, so v3 readers are not required to read v2 files. I like this because it is a great example of something out of scope. (Also note that forward-compatibility breaks are also done at a major version bump.) We don't need to decide how to handle backward compatibility. Our focus and need is to introduce forward-incompatible changes in a way that people can easily understand. > In Parquet-land, if we choose to bump the major version every time a new > encoding is added (a forward-incompatible change), this means we don't > have a way to signal backwards-incompatible changes such as a new file > footer. I disagree with this. First, I don't see any restriction on using a major version to drop backward compatibility. As you pointed out, Zarr did this and said v3 readers aren't required to be able to read v2 files. Second, changes can always be made in a way that is backward compatible. That is, new readers can detect and read old versions. Dropping backward compatibility is mostly about cleaning up mostly-dead code. > I'm not saying that Parquet shouldn't choose this scheme, but it's not > as obvious and widely-held as you're making it to be. For example we > could decide that forwards-incompatible changes are a minor version > bump, and backwards-incompatible changes a major version bump. Which is > actually closer to how SemVer works. I'm going to simplify by glossing over the differences between source and binary compatibility; the ideas are similar enough. In SemVer, a major version bump means my existing code might break because an API was removed or modified. A minor version bump means that new APIs were added but existing code will continue to work. So: - Major version change: existing code may break (this number defines a compatibility level) - Other version changes: existing code should not break The definition for formats is similar: - Major version change: existing readers may break (this number defines a compatibility level) - Other changes: existing readers continue to read correctly We can decide if we want to have minor versions or patch versions, but the core idea is the same. On Wed, Jul 8, 2026 at 4:50 PM Ryan Blue <[email protected]> wrote: > Antoine asked: > > what does "a community vote to close and adopt each major version": is > it about choosing when we freeze the version? or about which pending > features we decide to include in it? are we envisioning a situation where > we defer a feature to version N+1 for some reasons? > > Think of this vote just like a normal release vote. The specifics you're > asking about aren't determined up front. The community will discuss and > decide and come to consensus about it. > > We make decisions like these all the time for code releases: do we use a > release branch or main? When do we "freeze" to create a branch? How do we > handle late-arriving features? We may have written guidelines for these, > but it still comes down to building consensus when we know all the details. > > The guideline that we use in the Iceberg community (and used to in > Parquet) is that new forward-incompatible changes go into the next spec > version. We vote to add them, work on implementations to validate, and then > vote to close and adopt a version. Closing means that no additional > forward-incompatible changes will be made, and adopting basically means > that we all think the changes are correct enough to support over time (we > have not broken backward-compatibility). > > There have been times when we have pushed out features that are not > finished. Relative paths is a good example of one that we said we wanted in > v3, but in the end pushed out. We had a lot of other changes to release and > it was not close enough to completion that we thought it was a good idea to > wait. > > Discussion and consensus building work well for format definitions. We can > come up with scenarios where there are difficult choices, but just like > code releases, we can make it through those decisions with good faith and > reasonable discussion. > > On Wed, Jul 8, 2026 at 4:29 PM Ryan Blue <[email protected]> wrote: > >> In the ongoing vote thread about using versions to release >> forward-incompatible changes, there are a few questions to clarify how >> versions work in practice. >> >> I'm starting this thread to reply to those questions without polluting >> the vote thread with discussion and making it hard to follow. >> >> Please ask questions here! >> >> Gunnar asked: >> > Where do forward-compatible changes live in the proposed model? >> > Say, we're at v1.0 right now, and a non-breaking change gets added. Which >> version of the format represents that? >> >> Forward-compatible changes are documented in the spec, just like >> forward-incompatible changes. The difference is that you don't have to >> change the compatibility level of a file by increasing the format version >> to write them. >> >> A good example of this was the addition of column indexes, which was a >> forward-compatible change. Column indexes added two optional fields to >> `ColumnChunk` that older readers ignore, but that newer readers can use to >> find and deserialize `ColumnIndex` data located before the footer. The new >> fields and the new `ColumnIndex` structure were added to the thrift >> definition with no restriction about which version a file must be using in >> order to write them. >> >> I think this is confusing because of how we maintain the thrift file as >> the spec, but also supplement it with other markdown files. I mentioned >> this in the last Parquet sync, but I think it would help for us to have a >> markdown spec instead of using the thrift file, and to be clear about what >> docs are considered part of the spec. >> >> Another example: We used to use an enum for logical type but realized >> that this was a forward-incompatible change because older readers could not >> deserialize unknown enum symbols. We changed to the current approach so >> that a new logical type created a new field and metadata struct instead, so >> that unknown types are ignored as unknown thrift fields. But we should not >> have relied on compatibility of the thrift definition to carry this >> information. I think we should state clearly that new logical types are not >> breaking changes and that readers must ignore unknown logical types. >> >> Ryan >> >
