In the ongoing vote thread about using versions to release forward-incompatible changes, there are a few questions to clarify how versions work in practice.
I'm starting this thread to reply to those questions without polluting the vote thread with discussion and making it hard to follow. Please ask questions here! Gunnar asked: > Where do forward-compatible changes live in the proposed model? > Say, we're at v1.0 right now, and a non-breaking change gets added. Which version of the format represents that? Forward-compatible changes are documented in the spec, just like forward-incompatible changes. The difference is that you don't have to change the compatibility level of a file by increasing the format version to write them. A good example of this was the addition of column indexes, which was a forward-compatible change. Column indexes added two optional fields to `ColumnChunk` that older readers ignore, but that newer readers can use to find and deserialize `ColumnIndex` data located before the footer. The new fields and the new `ColumnIndex` structure were added to the thrift definition with no restriction about which version a file must be using in order to write them. I think this is confusing because of how we maintain the thrift file as the spec, but also supplement it with other markdown files. I mentioned this in the last Parquet sync, but I think it would help for us to have a markdown spec instead of using the thrift file, and to be clear about what docs are considered part of the spec. Another example: We used to use an enum for logical type but realized that this was a forward-incompatible change because older readers could not deserialize unknown enum symbols. We changed to the current approach so that a new logical type created a new field and metadata struct instead, so that unknown types are ignored as unknown thrift fields. But we should not have relied on compatibility of the thrift definition to carry this information. I think we should state clearly that new logical types are not breaking changes and that readers must ignore unknown logical types. Ryan
