I presented this at the biweekly sync and I am proposing to add it to the doc. https://github.com/apache/parquet-site/pull/220
On Thu, Oct 29, 2026 at 10:54 AM Julien Le Dem <[email protected]> wrote: > Here is the content of the doc > <https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a> > . Making changes to the format > Scope > > When making a change to the Parquet Format, we have to manage problems > that arise from the reader and the writer implementing a different version > of the spec. > > This document defines categories of changes to the Parquet Format, and > proposes a set of criteria to evaluate future potential changes. > Definitions > > A change to the parquet format can have the following properties: > > Compatibility: We narrow this down to producing the exact same tabular > data while possibly ignoring secondary data structures (like bloom filter, > indices, stats, …). Compatible implies Correct. > > - > > Backward compatibility: new readers can read files written by older > writers. > - > > Forward compatibility: old readers can read files written by newer > writers. > > Correctness: Not unknowingly reading data incorrectly > > - > > Incorrect read: a reader producing different data than the writer > intended. > - > > Correct failed read: a reader producing an error when it can not read > the data in the file. > > Partial Compatibility: when the reader can read a subset of the file (and > knows which subset) > > - > > If only the subset of columns supported is accessed, then read is > successful. > - > > The reader can elect to read only the subset of the data it > understands depending on the use case. > > [image: Screenshot 2026-10-29 at 10.50.18 AM.png]Dealing with > incompatible changesBackward compatibility > > When making a change to the spec, the new readers are aware of all the > past specs and therefore backward compatibility is usually required. > > Occasionally, we may drop support of a feature for technical reasons (ex: > dropping LZO support for read because of the additional complexity of > linking libraries). This is usually for features that are practically not > used anymore (or close). In such a case, new readers will fail with a clear > error and not produce correctness issues. > > In conclusion, a change is either always backward compatible or > occasionally backward incompatible and causes failure with an error and no > correctness issues. > Forward compatibility > > Old readers, however, are not aware of future spec changes. > Reader behavior > > ordered by desirability > > Name > > Possible behaviors of the reader > > When used > > Example > > User experience > > Correct Read > > Reads data correctly. > Sometimes ignores supplemental data structures it doesn’t know about. > > This is the most frequent case for forward compatibility. We add new > features that do not prevent old readers from reading the data correctly > even though they might not take advantage of those new features. > > You can read the data correctly and not understand bloom filters or > statistics > > Forward compatible: The format evolves seamlessly > > Correct Partial Read > > Reads correctly a subset of the columns as long as the unsupported columns > don’t need to be accessed and correctly fails otherwise. > > When a feature cannot be introduced without breaking forward compatibility > (ex: new encoding), old readers are able to read files and columns that do > not use that new feature and know they can not read the ones that do and > why. > > ALP: Old readers can read all the columns that don’t use this encoding > that applies only to floating point columns. > > Forward incompatible but correct: Old readers will explain failed reads > and preserve correctness. > > New writers can be configured to hold off on using new features until all > readers support it. > > Correct Failure to Read > > Can’t read data at all but understands why. > > When a feature cannot be introduced without breaking file-level > compatibility, old readers know they can not read the file. > > Changing the footer format. The reader has no granular information and > fails on the magic number change. > > Best Effort Read > > Reads the subset of data it understands and ignores the rest. > > Similar to Correct Partial Reads. In cases where this does not lead to > data loss, readers can elect to read only the subset of the file they can > read correctly. This is different from a correct partial read as a “select > *” would return only the columns it understands instead of failing for not > reading a subset of the columns. > > “Select *” Ignoring columns encoded in ALP and returning the subset of the > file that is readable. > > Forward incompatible best effort: Old readers can elect to read in best > effort the subset of the data. Not reading incorrect data but would lead to > dropping data if the output was saved without understanding the difference > with “correct partial read” > > Incorrect Read > > Reads data but unknowingly misinterprets it (ex: semantic changes to an > existing field like stats) > > When an old reader reads incorrect data because a semantic change is > indistinguishable and they have no way to know their read is incorrect. > This should never happen > > We avoid this case at all costs. > > Incorrect: Silently reading the data incorrectly. > Proposal: Evaluation system for changes to Parquet > > When making a change to the format we optimize for that user experience. > > Making changes to the Parquet format falls in 3 categories. > > - > > Forward compatible: Correct Read > - > > Forward incompatible: > - > > The metadata describes what can be read and not read: The reader > decides between Correct Partial Read and Best Effort Read > - > > The metadata doesn’t describe what can be read and not read (this > happens for example with the new footer format): Correct Failure to > Read > - > > We don’t make semantic changes that would cause old readers to > produce an Incorrect Read. > > [image: Screenshot 2026-10-29 at 10.51.50 AM.png] > > In particular that means that we avoid changing the semantics of an > existing field. For example changing a sort order that changes the min-max > statistics. Instead it can be made forward compatible by adding new > metadata fields. Making a forward compatible change like this is usually > preferable to breaking > > > On Wed, Oct 28, 2026 at 10:34 AM Julien Le Dem <[email protected]> wrote: > >> Following the thread that Alkis started >> <https://lists.apache.org/thread.html/qk8hxpvycy6161lqop1bynr3yt028m7t>(Thank >> you Alkis!) and the follow up conversations in the sync and otherwise, I >> have added a tab to the doc that formalizes how we categorize changes to >> the format, the corresponding impact on the user experience and how we >> minimize disruption. >> >> This is intentionally focusing on describing changes without going >> further (yet) in the implementation discussion. I'd like for us to converge >> on that first before we resume the implementation discussion (previous tab >> in the same doc) >> >> I think this should give us a solid common ground to resume the >> conversation. Please comment in the doc here: >> >> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a >> >
