Hello Ryan,
This is becoming very difficult to follow. It seems you're referring to
things that were discussed in private (or at least not on this list). It
also seems to be straying away from the original versioning discussion
to something else (magic bytes? why?).
I would strongly recommend that this discussion be made email-first.
Perhaps the Parquet community needs more explicit guidelines around
discussing spec additions/changes, or perhaps even a better process
altogether?
Regards
Antoine.
Le 01/09/2026 à 02:08, Ryan Blue a écrit :
I want to follow up on the open points here.
First, on the open question about magic bytes, I've confirmed with the
folks proposing the modular footer that we think we can handle encryption
within the footer. The reason for using the magic bytes for this before
(PARE) was that we needed to encrypt the entire footer to not leak
unencrypted data values. But with the modular design, we can leave the
first module of the footer unencrypted because it just contains offsets
that aren't sensitive and store a flag to indicate that the rest of the
modules are encrypted.
I think it still makes sense to use the magic bytes to encode the
Parquet format version, so that the entire footer content is covered by the
version. That avoids having two versions, one for the format and one for
the first footer module. I propose incrementing the last byte of the magic
bytes, starting with the next version we release. That is, if we release v3
next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet v10,
we could continue this pattern with 0x3A or skip to `PARA`... but this
decision would be a long time from now. In the meantime, we'd have one
version that is clearly visible from the magic bytes.
Second, for the question about whether to list preview features, I think
that we _should_ list the preview feature names in the footer. Extra
information is rarely harmful. I think the strongest argument against
listing the preview features is that implementations can use this
information to fail without a best-effort read. I think that a strict or
pedantic mode is a bad idea, but I prefer that it is an implementation
decision. There is going to be plenty of pressure not to fail needlessly,
so we don't need to encourage it at a format level.
Any other perspectives on these open issues?
Ryan
On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <[email protected]> wrote:
Sorry, I initially sent this through the wrong account that gets dropped
by ASF lists. Here's my summary from our conversation on Wednesday...
We discussed the open questions that Julien summarized above in the sync
and had agreement on a couple points, but there are also some open areas
for further discussion.
Broad agreement:
- *For Q1 and Q2, we choose option B*. We want consistency across
implementations, but cannot dictate behaviors like specific API methods or
reader modes.
- We will define consistent names or labels for preview features to help
consistency
- *For Q3, we eliminated option B*: We do not want to indicate preview
features without saying what those preview features are
- Q3 covers two separate questions: magic bytes and how to track preview
features (see open questions)
- *For Q4, we choose option C*: it is an implementation decision to have
a strict mode
Open questions:
- Do we want to add the format version to the magic bytes, OR change magic
bytes only when the footer encoding changes?
- Do we want to list the preview features that are used in a file, OR do
we omit them because we don't want to encourage unnecessary failures?
Here's a more detailed summary:
For 1 and 2, there was general consensus that we want to have consistency
across implementations, but it doesn't make sense to mandate writer API
options in a file format's spec that covers on-disk representation. We
agreed that we want to define labels for preview features to help ensure
consistency (and these may be stored in the format). For example, we could
agree and document that "alp-encoding" means the new ALP encoding for float
and double values, so that people can refer to the feature consistently,
even if the API to enable features is idiomatic and slightly different
across languages. We also clarified the requirement that preview features
must cause readers that do not implement support to fail; readers must not
produce incorrect results.
Micah also suggested that we introduce minor versions (which must be
forward-compatible within a version) and allow pinning to those minor
versions. This would prevent using newer features even though they do not
affect correctness. There wasn't an example when this would be useful and
we concluded that this can be done later if we discover a use case.
For 3, we discussed that there are two decision points. 3.1 is how the
magic bytes relate to the format version, and 3.2 is whether we want the
preview feature names included in files when they are used. (Option 2 was
discarded; it was for, essentially, a boolean flag to indicate preview
features were used but did not specify which ones.)
In the discussion about magic bytes (3.1), we outlined two options: we can
encode the version in the magic bytes and update it each time the format
version changes, or we can store the version in the footer and use a
separate version in the magic bytes that is incremented when the footer
representation changes ("PARA", "PARB", etc.). If we store the format
version in the footer, we would need to stop using "PAR1" and "PARE"
because we would want to be able to encode more than versions 3-9 and have
to handle encryption.
The discussion about preview features relates to question 4, which is
whether we want to require readers to expose a "pedantic" mode (think:
strict). Several people were advocating that readers should always attempt
to read (best effort), and therefore we don't need to make changes to
enable a strict mode like keeping a list of preview feature names. If
implementations choose to have a pedantic mode, they can easily detect
changes like unsupported encodings in columns that are not projected and
fail with a good error message. We agreed that pedantic mode is a decision
for the read implementation (outside the file format spec's scope), but did
not conclude that we didn't want the list of preview features.
On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <[email protected]> wrote:
Following up on the discussion during the Parquet sync on Wednesday, I
have
added a new tab to the Versioning document
<
https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
to discuss 4 remaining decision points for the implementation. For each I
have listed options and pros and cons.
1. Mechanism to decide what Parquet features writers should use
2. Mechanism to write Parquet files that use a preview feature
3. Mechanism to record in the file what features it uses
4. Describing reader behavior when reading a file that is not fully
supported
The stated guiding principle is: We want to minimize prescribing how
readers and writers should work and focus on giving them the information
to
implement the behavior they deem appropriate.
Please comment in the doc to help finalize consensus.
(thank you Andrew for early feedback and helping me think this through!)
