Antoine, The Versioning doc Ryan is referring to is this one:
https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf It was sent to this list on July 9 ( https://www.mail-archive.com/[email protected]/msg27611.html), and what we're discussing now is the latest tab. Magic bytes are item 3 in that tab, "Mechanism to record in the file what features it uses," Option D. So this isn't a new topic and it isn't one from a private channel. I agree that decisions need to happen on this list which is why I appreciate Ryan's email. He summarized the public weekly sync and posted it here so that people who weren't in the sync can weigh in, explicitly calling out the open questions as still open. Nothing has been decided and folks should feel free to re-open questions on other things on the documents if they like although I would urge folks to try to find consensus when possible. I'd rather keep this thread focused on the design and try to stay away from meta-discussions about process. If there is anything we need to discuss from the document we should probably bring it up here and go into more depth. On Wed, Sep 2, 2026 at 10:58 AM Russell Spitzer <[email protected]> wrote: > Micah, > > I do not think 1 and 2 are contentious; they follow from the vote. > > 1. A file that uses a preview feature must be readable by any other reader > that implements that preview, and by readers of the major version that > later adopts it. > > 2. A reader that does not support a preview feature must not return > incorrect data. If existing readers would silently misread it, it is not a > preview feature; it belongs in the next major version. > > I would draw 3 and 4 a bit differently. > > 3. If a reader cannot parse the Thrift footer, it does not support that > version of Parquet. For everything else (preview features), a detailed > error is fine. > > 4. This mixes version and preview. A V2 reader should fail fast on a file > labeled V3; we cannot guarantee a correct read even if it undrestands some > of the structure. A V2 reader should be able to safely try to read a V2 > file that uses preview features it does not implement. It should only fail > if it reaches a feature it does not understand and cannot skip (Example - > Encoding, Logical Type, ...) > > Russ > > > > > On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <[email protected]> wrote: > >> >> Hello Ryan, >> >> This is becoming very difficult to follow. It seems you're referring to >> things that were discussed in private (or at least not on this list). It >> also seems to be straying away from the original versioning discussion >> to something else (magic bytes? why?). >> >> I would strongly recommend that this discussion be made email-first. >> Perhaps the Parquet community needs more explicit guidelines around >> discussing spec additions/changes, or perhaps even a better process >> altogether? >> >> Regards >> >> Antoine. >> >> >> Le 01/09/2026 à 02:08, Ryan Blue a écrit : >> > I want to follow up on the open points here. >> > >> > First, on the open question about magic bytes, I've confirmed with the >> > folks proposing the modular footer that we think we can handle >> encryption >> > within the footer. The reason for using the magic bytes for this before >> > (PARE) was that we needed to encrypt the entire footer to not leak >> > unencrypted data values. But with the modular design, we can leave the >> > first module of the footer unencrypted because it just contains offsets >> > that aren't sensitive and store a flag to indicate that the rest of the >> > modules are encrypted. >> > >> > I think it still makes sense to use the magic bytes to encode the >> > Parquet format version, so that the entire footer content is covered by >> the >> > version. That avoids having two versions, one for the format and one for >> > the first footer module. I propose incrementing the last byte of the >> magic >> > bytes, starting with the next version we release. That is, if we >> release v3 >> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet >> v10, >> > we could continue this pattern with 0x3A or skip to `PARA`... but this >> > decision would be a long time from now. In the meantime, we'd have one >> > version that is clearly visible from the magic bytes. >> > >> > Second, for the question about whether to list preview features, I think >> > that we _should_ list the preview feature names in the footer. Extra >> > information is rarely harmful. I think the strongest argument against >> > listing the preview features is that implementations can use this >> > information to fail without a best-effort read. I think that a strict or >> > pedantic mode is a bad idea, but I prefer that it is an implementation >> > decision. There is going to be plenty of pressure not to fail >> needlessly, >> > so we don't need to encourage it at a format level. >> > >> > Any other perspectives on these open issues? >> > >> > Ryan >> > >> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <[email protected]> wrote: >> > >> >> Sorry, I initially sent this through the wrong account that gets >> dropped >> >> by ASF lists. Here's my summary from our conversation on Wednesday... >> >> >> >> We discussed the open questions that Julien summarized above in the >> sync >> >> and had agreement on a couple points, but there are also some open >> areas >> >> for further discussion. >> >> >> >> Broad agreement: >> >> - *For Q1 and Q2, we choose option B*. We want consistency across >> >> implementations, but cannot dictate behaviors like specific API >> methods or >> >> reader modes. >> >> - We will define consistent names or labels for preview features to >> help >> >> consistency >> >> - *For Q3, we eliminated option B*: We do not want to indicate preview >> >> features without saying what those preview features are >> >> - Q3 covers two separate questions: magic bytes and how to track >> preview >> >> features (see open questions) >> >> - *For Q4, we choose option C*: it is an implementation decision to >> have >> >> a strict mode >> >> >> >> Open questions: >> >> - Do we want to add the format version to the magic bytes, OR change >> magic >> >> bytes only when the footer encoding changes? >> >> - Do we want to list the preview features that are used in a file, OR >> do >> >> we omit them because we don't want to encourage unnecessary failures? >> >> >> >> Here's a more detailed summary: >> >> >> >> For 1 and 2, there was general consensus that we want to have >> consistency >> >> across implementations, but it doesn't make sense to mandate writer API >> >> options in a file format's spec that covers on-disk representation. We >> >> agreed that we want to define labels for preview features to help >> ensure >> >> consistency (and these may be stored in the format). For example, we >> could >> >> agree and document that "alp-encoding" means the new ALP encoding for >> float >> >> and double values, so that people can refer to the feature >> consistently, >> >> even if the API to enable features is idiomatic and slightly different >> >> across languages. We also clarified the requirement that preview >> features >> >> must cause readers that do not implement support to fail; readers must >> not >> >> produce incorrect results. >> >> >> >> Micah also suggested that we introduce minor versions (which must be >> >> forward-compatible within a version) and allow pinning to those minor >> >> versions. This would prevent using newer features even though they do >> not >> >> affect correctness. There wasn't an example when this would be useful >> and >> >> we concluded that this can be done later if we discover a use case. >> >> >> >> For 3, we discussed that there are two decision points. 3.1 is how the >> >> magic bytes relate to the format version, and 3.2 is whether we want >> the >> >> preview feature names included in files when they are used. (Option 2 >> was >> >> discarded; it was for, essentially, a boolean flag to indicate preview >> >> features were used but did not specify which ones.) >> >> >> >> In the discussion about magic bytes (3.1), we outlined two options: we >> can >> >> encode the version in the magic bytes and update it each time the >> format >> >> version changes, or we can store the version in the footer and use a >> >> separate version in the magic bytes that is incremented when the footer >> >> representation changes ("PARA", "PARB", etc.). If we store the format >> >> version in the footer, we would need to stop using "PAR1" and "PARE" >> >> because we would want to be able to encode more than versions 3-9 and >> have >> >> to handle encryption. >> >> >> >> The discussion about preview features relates to question 4, which is >> >> whether we want to require readers to expose a "pedantic" mode (think: >> >> strict). Several people were advocating that readers should always >> attempt >> >> to read (best effort), and therefore we don't need to make changes to >> >> enable a strict mode like keeping a list of preview feature names. If >> >> implementations choose to have a pedantic mode, they can easily detect >> >> changes like unsupported encodings in columns that are not projected >> and >> >> fail with a good error message. We agreed that pedantic mode is a >> decision >> >> for the read implementation (outside the file format spec's scope), >> but did >> >> not conclude that we didn't want the list of preview features. >> >> >> >> >> >> >> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <[email protected]> >> wrote: >> >> >> >>> Following up on the discussion during the Parquet sync on Wednesday, I >> >>> have >> >>> added a new tab to the Versioning document >> >>> < >> >>> >> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf >> >>>> >> >>> to discuss 4 remaining decision points for the implementation. For >> each I >> >>> have listed options and pros and cons. >> >>> 1. Mechanism to decide what Parquet features writers should use >> >>> 2. Mechanism to write Parquet files that use a preview feature >> >>> 3. Mechanism to record in the file what features it uses >> >>> 4. Describing reader behavior when reading a file that is not fully >> >>> supported >> >>> >> >>> The stated guiding principle is: We want to minimize prescribing how >> >>> readers and writers should work and focus on giving them the >> information >> >>> to >> >>> implement the behavior they deem appropriate. >> >>> >> >>> Please comment in the doc to help finalize consensus. >> >>> (thank you Andrew for early feedback and helping me think this >> through!) >> >>> >> >> >> > >> >> >>
