I don't see how any option except #1, "fail on an unsupported version" is realistic. As "pretending you support a version you don't understand" is simply a pretence.
The fact that a parquet lib is required to fail fast on an unknown version is a strong argument against moving to a new version unless you have a really, really good reason. the entire work for having flatbuf alongside the thrift buffers is to deliver performance improvements while retaining compatibility. I would argue for trying to go that way as long as possible At the same time as that "as long as possible" clause, I could imagine a parquet release "2.0" which can generate and understand classic parquet + with flatbuff & thrift, or parquet without the thrift and maybe with some other optimisations. By default it' might generate parquet 1.x files, but it would be aware of the later parquet version so *would* be forward compatible. steve (originally sent from my gmail account; should have been blocked, apologies if it does actually get through and so arrives twice) On Wed, 16 Sept 2026 at 20:56, Fokko Driesprong <[email protected]> wrote: > Sorry, I had to catch up here. I'm in favor of 1. > > Proposed proper breaking changes, like the new footer, will always break > existing readers. I don't think we can handle this gracefully with option > ≥2. > > For breaking changes, we should fail because we cannot control how the > client will behave. Also, we don't control all implementations out there, > so we can't make assumptions about whether they use a field. The only thing > we control is the specification. > > The path-in-schema sounds like a harmless change, but I think it opens the > door to a slippery slope of further breaking changes that will cause > existing readers to break more and more, and folks will lose trust in the > project. Like Xiening is saying, #1 is the clean option. Instead of cutting > corners, we could make the path-in-schema optimization part of the new > footer. Bundle all of this in a new version. > > Kind regards, > Fokko > > On 2026/09/15 18:26:22 Xiening Dai wrote: > > Short answer - I vote for option 1. > > > > I think what you are talking about here for option 1 and 2 are regarding > the Apache Parquet library, or Parquet official spec definition. We cannot > prevent anyone from implementing their own Parquet reader (and in fact I > know a few engines do have their own implementation). So #3 is already a > given for anything outside of Apache Parquet. > > > > Now when comes to option 1 and 2. I believe #1 is clean. A full stop > provides well defined behavior. #2 is going to complicate the reader > implementation. It's hard to test, hard to define the behavior, and hard > for user to understand and predict the results. On the surface, the reader > can potential consume more files, but the extra complexity and ambiguities > makes it less appealing. > > > > In addition to this, I've seen a proposal to change Parquet magic number > to achieve option #1. I don't support such approach. There are many > tools/scripts rely on that particular magic number to detect file type, > such as s3 object crawler, etc., and they don't even need to be a reader. > And they will be broken if we change magic number. I think using the file > version should be good enough. We should just release a reader patch to > fail fast when version is higher than what's supported. And this patch > should be released before we upgrade Parquet version and introduce any > breaking change. > > > > On 2026/09/09 22:38:19 Ryan Blue wrote: > > > Hi everyone, > > > > > > This is a new thread to discuss an issue raised in the "Finalizing the > > > versioning proposal" thread. I suggested starting a new thread to > focus on > > > this question in the community sync and there was agreement so that we > can > > > get more attention to the issue. > > > > > > The question that we identified is: How should a reader behave when it > > > encounters a file written with an unsupported format version? > > > > > > There are 3 main options: > > > 1. A reader should fail because it does not support the version > > > 2. A reader should attempt to read the file > > > 3. This choice is left up to implementations > > > > > > I'll cover each option in more detail below, but first I want to > clarify > > > that we are not talking about "preview" features like encodings or > > > forward-compatible changes like new logical types. Preview features > will > > > break readers that do not support them and only affect specific columns > > > using the feature. For preview features, the expectation is that > readers > > > will attempt to read the file and will fail if they need to project a > > > column that cannot be read. > > > > > > The choice of how to handle an unsupported format version primarily > affects > > > changes that add, remove, or modify the semantics of metadata fields. > For > > > example: > > > - Changing `path_in_schema` from required to optional > > > - Adding `offset` and `size` fields to a page header to relocate page > data > > > - Fixing stats written with the wrong sort order > > > > > > If we choose option 1, then a v3 reader that encounters a v4 file > would be > > > required to fail with an error that the file is too new to be read. > With > > > the current versioning scheme, the forward-compatibility within a > version > > > means that a v3 reader can read any v3 file and will produce the > correct > > > data. But this guarantee does not extend to any higher version. > Without a > > > guarantee that results are correct, the v3 reader must reject a v4 > file. > > > > > > If we choose option 2, then the v3 reader will attempt to read the v4 > file. > > > In this case, Parquet will need to provide an additional guarantee that > > > reading future formats will either fail or produce correct results. If > I > > > understand correctly, this option would use Thrift's semantics to cause > > > failures. For example, path_in_schema would be required in Thrift and > > > readers are expected to fail in decoding. > > > > > > Option 3 would mean that readers may choose to attempt to read, but do > not > > > have additional guarantees. > > > > > > Hopefully this makes the choices clear and we can have a discussion on > this > > > thread. Let's try to keep this thread on the topic of these options, > either > > > to clarify them or to debate the effects. We can discuss process or > other > > > topics on new threads. > > > > > > Ryan > > > > > >
