I'll start off discussion on this thread in a separate email so my opinion is separate from the context for this discussion where I was staying neutral.
I strongly support option 1, that readers must fail when reading a file written with an unsupported format version. The trade-off between these options comes down to the additional guarantee needed for option 2, that readers will either read correctly or fail. The scope of this guarantee is impractical, for two reasons. First, the scope includes all implementations and all future versions. This huge surface area will inevitably lead to bad assumptions and correctness bugs. Second, this guarantee would severely limit Parquet's ability to evolve as a format. Every metadata change would be more expensive to design and validate. In the Parquet sync, Will asked why Thrift's evolution rules aren't enough. I think path_in_schema is a good example of why option 2 will inevitably produce correctness bugs. For anyone not familiar, path_in_schema is a string list in each ColumnMetaData that contains the column path, like ["request", "url"]. Many implementations don't need this path and we would like to make the footer lighter by omitting it. The rationale is that when Thrift encounters a required field that is missing, it should throw an exception. This assumes the error behavior is universal -- and we don't know that. I know of at least two custom Thrift implementations that were built for faster deserialization. I don't know how those implementations deserialize required fields. Maybe they assume path_in_schema is always present because of a writer guarantee that has been true up until now? Couldn't you assume that if it's required, then the writer will always produce it? We also cannot necessarily rely on the input to Thrift marking the field required. Some projects automatically pull the thrift definition from parquet-format (including Parquet Java until recently). If an implementation updated its Thrift definition, it could easily pull in a new version in which path_in_schema is optional and the read-time validation we expect from Thrift would no longer happen. From there, it could easily lead to a correctness bug if the implementation used utilities like Java's HashMap and Apache Commons StringUtils, which are both null-tolerant: StringUtils.join(null, ".") produces null, and HashMap works fine with null keys. Unexpected null paths could lead to missing columns when accessed by name. I'll note that Will objected to the path_in_schema example in the sync because implementations "should not" sync the Thrift definition like this. But I don't think that's true. Parquet Java would sync the Thrift file <https://github.com/apache/parquet-java/pull/3611> as I'm describing. The writer requirement to continue writing path_in_schema kept implementations safe, not the Thrift definition. The safe option is to not make a breaking change to omit the field, but I think we all agree that we need to be able to make this type of change. My second argument for why we should not attempt to read all future versions of Parquet is that it ends up limiting how we can evolve the format. I think one of the most important outcomes of this versioning discussion is the ability to move more quickly and update the format. Adding a guarantee that all older readers will continue to read correctly or will fail means that we can never change the meaning of a field or add fields that change how a structure is interpreted. A quick example is path_in_schema again, where I think the safe option is to continue writing the field forever. Another example is a discussion related to Dan's proposal for non-contiguous pages (not a single chunk per column per row group). One of the alternatives we have brought up internally is still using the same page layout, but being able to point to 1page data elsewhere in the file by adding `offset` and `size` fields to the page header. If we were to add `offset` and `size` to a page header, we would change the meaning of the header. Readers that don't know about these two new fields would ignore them (hm, probably?). These older readers would not know the content was relocated and could start decoding the bits directly after the header. That's not good, so if we are making a guarantee that these readers should never produce bad data, then we should instead add a new symbol to the PageType, DATA_PAGE_V3, and copy all the fields from the previous page header. But we should only use DATA_PAGE_V3 for pages that require the `offset` and `size`, or else existing readers can't read all of the non-relocated columns. Now we would have 3 page types, along with the code to handle each one <https://github.com/apache/parquet-java/pull/3611>. Each time we want to add columns that affect how a metadata struct is interpreted or slightly change the meaning of a column, we have to add a copy instead in order to guarantee that the old one is available, unchanged, for older readers. This guarantee would mean we can't move on from past mistakes (like PARQUET-251 or other stats bugs) and would inflate the size of metadata. I think a better idea is the alternative, which is to break reader compatibility at each new format version. When a file is targeted for a new format version, we can remove path_in_schema and we can add new fields and reuse existing structures. I think the project will move faster if we do this and will be more reliable because we won't have to worry about the behavior of unknown implementations. We state required behavior for a new version and then we can rely on it. I know there's also a concern about how to move between format releases. This is a legitimate concern. I think the solution is to choose what is forward-compatible, what goes into preview features and can be used early, and what goes into a breaking format change. If at all possible we should make changes forward-compatible so that we can add them to the current format version. New features like encodings should be released as preview features so they can be enabled early. People shouldn't have to weigh a breaking format version against new features if possible. If we do those, then hopefully we can make changes in a new format version small and easy to support, like the changes for handling a missing path_in_schema field. Ryan On Wed, Sep 9, 2026 at 3:38 PM Ryan Blue <[email protected]> wrote: > Hi everyone, > > This is a new thread to discuss an issue raised in the "Finalizing the > versioning proposal" thread. I suggested starting a new thread to focus on > this question in the community sync and there was agreement so that we can > get more attention to the issue. > > The question that we identified is: How should a reader behave when it > encounters a file written with an unsupported format version? > > There are 3 main options: > 1. A reader should fail because it does not support the version > 2. A reader should attempt to read the file > 3. This choice is left up to implementations > > I'll cover each option in more detail below, but first I want to clarify > that we are not talking about "preview" features like encodings or > forward-compatible changes like new logical types. Preview features will > break readers that do not support them and only affect specific columns > using the feature. For preview features, the expectation is that readers > will attempt to read the file and will fail if they need to project a > column that cannot be read. > > The choice of how to handle an unsupported format version primarily > affects changes that add, remove, or modify the semantics of metadata > fields. For example: > - Changing `path_in_schema` from required to optional > - Adding `offset` and `size` fields to a page header to relocate page data > - Fixing stats written with the wrong sort order > > If we choose option 1, then a v3 reader that encounters a v4 file would be > required to fail with an error that the file is too new to be read. With > the current versioning scheme, the forward-compatibility within a version > means that a v3 reader can read any v3 file and will produce the correct > data. But this guarantee does not extend to any higher version. Without a > guarantee that results are correct, the v3 reader must reject a v4 file. > > If we choose option 2, then the v3 reader will attempt to read the v4 > file. In this case, Parquet will need to provide an additional guarantee > that reading future formats will either fail or produce correct results. If > I understand correctly, this option would use Thrift's semantics to cause > failures. For example, path_in_schema would be required in Thrift and > readers are expected to fail in decoding. > > Option 3 would mean that readers may choose to attempt to read, but do not > have additional guarantees. > > Hopefully this makes the choices clear and we can have a discussion on > this thread. Let's try to keep this thread on the topic of these options, > either to clarify them or to debate the effects. We can discuss process or > other topics on new threads. > > Ryan >
