Sorry, I had to catch up here. I'm in favor of 1. Proposed proper breaking changes, like the new footer, will always break existing readers. I don't think we can handle this gracefully with option ≥2.
For breaking changes, we should fail because we cannot control how the client will behave. Also, we don't control all implementations out there, so we can't make assumptions about whether they use a field. The only thing we control is the specification. The path-in-schema sounds like a harmless change, but I think it opens the door to a slippery slope of further breaking changes that will cause existing readers to break more and more, and folks will lose trust in the project. Like Xiening is saying, #1 is the clean option. Instead of cutting corners, we could make the path-in-schema optimization part of the new footer. Bundle all of this in a new version. Kind regards, Fokko On 2026/09/15 18:26:22 Xiening Dai wrote: > Short answer - I vote for option 1. > > I think what you are talking about here for option 1 and 2 are regarding the > Apache Parquet library, or Parquet official spec definition. We cannot > prevent anyone from implementing their own Parquet reader (and in fact I know > a few engines do have their own implementation). So #3 is already a given for > anything outside of Apache Parquet. > > Now when comes to option 1 and 2. I believe #1 is clean. A full stop provides > well defined behavior. #2 is going to complicate the reader implementation. > It's hard to test, hard to define the behavior, and hard for user to > understand and predict the results. On the surface, the reader can potential > consume more files, but the extra complexity and ambiguities makes it less > appealing. > > In addition to this, I've seen a proposal to change Parquet magic number to > achieve option #1. I don't support such approach. There are many > tools/scripts rely on that particular magic number to detect file type, such > as s3 object crawler, etc., and they don't even need to be a reader. And they > will be broken if we change magic number. I think using the file version > should be good enough. We should just release a reader patch to fail fast > when version is higher than what's supported. And this patch should be > released before we upgrade Parquet version and introduce any breaking change. > > On 2026/09/09 22:38:19 Ryan Blue wrote: > > Hi everyone, > > > > This is a new thread to discuss an issue raised in the "Finalizing the > > versioning proposal" thread. I suggested starting a new thread to focus on > > this question in the community sync and there was agreement so that we can > > get more attention to the issue. > > > > The question that we identified is: How should a reader behave when it > > encounters a file written with an unsupported format version? > > > > There are 3 main options: > > 1. A reader should fail because it does not support the version > > 2. A reader should attempt to read the file > > 3. This choice is left up to implementations > > > > I'll cover each option in more detail below, but first I want to clarify > > that we are not talking about "preview" features like encodings or > > forward-compatible changes like new logical types. Preview features will > > break readers that do not support them and only affect specific columns > > using the feature. For preview features, the expectation is that readers > > will attempt to read the file and will fail if they need to project a > > column that cannot be read. > > > > The choice of how to handle an unsupported format version primarily affects > > changes that add, remove, or modify the semantics of metadata fields. For > > example: > > - Changing `path_in_schema` from required to optional > > - Adding `offset` and `size` fields to a page header to relocate page data > > - Fixing stats written with the wrong sort order > > > > If we choose option 1, then a v3 reader that encounters a v4 file would be > > required to fail with an error that the file is too new to be read. With > > the current versioning scheme, the forward-compatibility within a version > > means that a v3 reader can read any v3 file and will produce the correct > > data. But this guarantee does not extend to any higher version. Without a > > guarantee that results are correct, the v3 reader must reject a v4 file. > > > > If we choose option 2, then the v3 reader will attempt to read the v4 file. > > In this case, Parquet will need to provide an additional guarantee that > > reading future formats will either fail or produce correct results. If I > > understand correctly, this option would use Thrift's semantics to cause > > failures. For example, path_in_schema would be required in Thrift and > > readers are expected to fail in decoding. > > > > Option 3 would mean that readers may choose to attempt to read, but do not > > have additional guarantees. > > > > Hopefully this makes the choices clear and we can have a discussion on this > > thread. Let's try to keep this thread on the topic of these options, either > > to clarify them or to debate the effects. We can discuss process or other > > topics on new threads. > > > > Ryan > > >
