Here is the content of the doc
<https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a>
. Making changes to the format
Scope
When making a change to the Parquet Format, we have to manage problems that
arise from the reader and the writer implementing a different version of
the spec.
This document defines categories of changes to the Parquet Format, and
proposes a set of criteria to evaluate future potential changes.
Definitions
A change to the parquet format can have the following properties:
Compatibility: We narrow this down to producing the exact same tabular data
while possibly ignoring secondary data structures (like bloom filter,
indices, stats, …). Compatible implies Correct.
-
Backward compatibility: new readers can read files written by older
writers.
-
Forward compatibility: old readers can read files written by newer
writers.
Correctness: Not unknowingly reading data incorrectly
-
Incorrect read: a reader producing different data than the writer
intended.
-
Correct failed read: a reader producing an error when it can not read
the data in the file.
Partial Compatibility: when the reader can read a subset of the file (and
knows which subset)
-
If only the subset of columns supported is accessed, then read is
successful.
-
The reader can elect to read only the subset of the data it understands
depending on the use case.
[image: Screenshot 2026-10-29 at 10.50.18 AM.png]Dealing with incompatible
changesBackward compatibility
When making a change to the spec, the new readers are aware of all the past
specs and therefore backward compatibility is usually required.
Occasionally, we may drop support of a feature for technical reasons (ex:
dropping LZO support for read because of the additional complexity of
linking libraries). This is usually for features that are practically not
used anymore (or close). In such a case, new readers will fail with a clear
error and not produce correctness issues.
In conclusion, a change is either always backward compatible or
occasionally backward incompatible and causes failure with an error and no
correctness issues.
Forward compatibility
Old readers, however, are not aware of future spec changes.
Reader behavior
ordered by desirability
Name
Possible behaviors of the reader
When used
Example
User experience
Correct Read
Reads data correctly.
Sometimes ignores supplemental data structures it doesn’t know about.
This is the most frequent case for forward compatibility. We add new
features that do not prevent old readers from reading the data correctly
even though they might not take advantage of those new features.
You can read the data correctly and not understand bloom filters or
statistics
Forward compatible: The format evolves seamlessly
Correct Partial Read
Reads correctly a subset of the columns as long as the unsupported columns
don’t need to be accessed and correctly fails otherwise.
When a feature cannot be introduced without breaking forward compatibility
(ex: new encoding), old readers are able to read files and columns that do
not use that new feature and know they can not read the ones that do and
why.
ALP: Old readers can read all the columns that don’t use this encoding that
applies only to floating point columns.
Forward incompatible but correct: Old readers will explain failed reads and
preserve correctness.
New writers can be configured to hold off on using new features until all
readers support it.
Correct Failure to Read
Can’t read data at all but understands why.
When a feature cannot be introduced without breaking file-level
compatibility, old readers know they can not read the file.
Changing the footer format. The reader has no granular information and
fails on the magic number change.
Best Effort Read
Reads the subset of data it understands and ignores the rest.
Similar to Correct Partial Reads. In cases where this does not lead to data
loss, readers can elect to read only the subset of the file they can read
correctly. This is different from a correct partial read as a “select *”
would return only the columns it understands instead of failing for not
reading a subset of the columns.
“Select *” Ignoring columns encoded in ALP and returning the subset of the
file that is readable.
Forward incompatible best effort: Old readers can elect to read in best
effort the subset of the data. Not reading incorrect data but would lead to
dropping data if the output was saved without understanding the difference
with “correct partial read”
Incorrect Read
Reads data but unknowingly misinterprets it (ex: semantic changes to an
existing field like stats)
When an old reader reads incorrect data because a semantic change is
indistinguishable and they have no way to know their read is incorrect.
This should never happen
We avoid this case at all costs.
Incorrect: Silently reading the data incorrectly.
Proposal: Evaluation system for changes to Parquet
When making a change to the format we optimize for that user experience.
Making changes to the Parquet format falls in 3 categories.
-
Forward compatible: Correct Read
-
Forward incompatible:
-
The metadata describes what can be read and not read: The reader
decides between Correct Partial Read and Best Effort Read
-
The metadata doesn’t describe what can be read and not read (this
happens for example with the new footer format): Correct Failure to
Read
-
We don’t make semantic changes that would cause old readers to
produce an Incorrect Read.
[image: Screenshot 2026-10-29 at 10.51.50 AM.png]
In particular that means that we avoid changing the semantics of an
existing field. For example changing a sort order that changes the min-max
statistics. Instead it can be made forward compatible by adding new
metadata fields. Making a forward compatible change like this is usually
preferable to breaking
On Wed, Oct 28, 2026 at 10:34 AM Julien Le Dem <[email protected]> wrote:
> Following the thread that Alkis started
> <https://lists.apache.org/thread.html/qk8hxpvycy6161lqop1bynr3yt028m7t>(Thank
> you Alkis!) and the follow up conversations in the sync and otherwise, I
> have added a tab to the doc that formalizes how we categorize changes to
> the format, the corresponding impact on the user experience and how we
> minimize disruption.
>
> This is intentionally focusing on describing changes without going further
> (yet) in the implementation discussion. I'd like for us to converge on that
> first before we resume the implementation discussion (previous tab in the
> same doc)
>
> I think this should give us a solid common ground to resume the
> conversation. Please comment in the doc here:
>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a
>
