Le 09/09/2026 à 02:10, Ryan Blue a écrit :
And finally I think there's a group of us (Dan, Alkis, and I) who would
prefer requiring readers to fail if they encounter a newer format version
rather than trying to read it. The main argument for this is to make
correctness guarantees. I don't think it is enough to rely on Thrift
evolution rules, and I think that attempting to will inevitably lead to
correctness bugs when our changes have unforeseen consequences in readers.
If such correctness bugs were likely, then surely they would have
happened already, given all the features we have added in the past?
Do we have any example of correctness bugs introduced in older readers
by the existence of a new feature?
> I think that relying on Thrift evolution rules also limits the ways
that we
> can evolve the format and make it significantly harder to implement and
> maintain.
I agree that changes that break *existing* semantics need some signaling
in the Parquet metadata. The problem is that we are conflating such
changes, with other changes that only introduce *new* semantics.
For example, the ALP encoding is introducing new semantics that won't be
understood by existing readers. This deserves specific guidance to the
user for when to produce files with ALP enabled (hence the idea of
compatibility levels, e.g. "please write a file with V3-level
features"). But the ALP encoding doesn't threaten interoperability for
whatever features a pre-ALP reader supports.
Most features we add to the Parquet format are similar to the ALP
encoding. They can be ignored by older readers.
Regards
Antoine.
