In my opinion, the core problem we need to solve here is how Parquet writers communicate to Parquet readers which features must be supported in order to guarantee they can completely read a given file.
I thought that we all basically agree that a version is the way to communicate this. The only open question I think is how to encode this version in the file: 1. The magic bytes in the file PAR1, PARE[1], etc. 2. The version field in the metadata[2]. I personally think either would work, and choosing either one would be an improvement over the status quo. Also, while it is important to work through other cases (e.g. communicating preview feature inclusion) I think we can treat that as a follow on project / vote as it will likely involve a separate mechanism. Andrew [1]: https://parquet.apache.org/docs/file-format/ [2]: https://github.com/apache/parquet-format/blob/a645b5a2cfa1453b49eef2aa0eec5bb4498a69cd/src/main/thrift/parquet.thrift#L1408-L1416 On Wed, Sep 2, 2026 at 12:42 PM Antoine Pitrou <[email protected]> wrote: > > """ > Readers need a simple mechanism to know whether they support all > features they may encounter when reading a specific file. > """ > > I would like to dispute this. No argument is given as to why readers > need something else than what is already embodied in the metadata. > > It's also not obvious why that information is granular enough (some > features are optional, the user may not want to read all columns anyway, > etc.). > > At least in Arrow, I don't remember a single user complaining that they > don't have a simpler access to that information. > > Regards > > Antoine. > > > > Le 14/08/2026 à 20:25, Julien Le Dem a écrit : > > Following up on the discussion during the Parquet sync on Wednesday, I > have > > added a new tab to the Versioning document > > < > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > > > > to discuss 4 remaining decision points for the implementation. For each I > > have listed options and pros and cons. > > 1. Mechanism to decide what Parquet features writers should use > > 2. Mechanism to write Parquet files that use a preview feature > > 3. Mechanism to record in the file what features it uses > > 4. Describing reader behavior when reading a file that is not fully > > supported > > > > The stated guiding principle is: We want to minimize prescribing how > > readers and writers should work and focus on giving them the information > to > > implement the behavior they deem appropriate. > > > > Please comment in the doc to help finalize consensus. > > (thank you Andrew for early feedback and helping me think this through!) > > >
