> > It worries me if the version number in the footer describes the features used in the file generally, rather than only versioning the footer protocol.
Versioning the footer format itself allows us to introduce a breaking change. If we do anything in the footer that means that unaware readers would mis-parse the footer then we’d bump the version number. (Making a required field optional is a breaking change. And proposals for breaking footers would always start their new footers with a tiny thrift object with the version number. Almost everything else can always be introduced in a non-breaking way.) If the footer isn’t itself versioned then there is no way to introduce breaking changes that impact the footer itself. Whereas if the version in the footer is saying that there is some features of that future version somewhere in the file, that doesn't help a reader who doesn’t know what that version is to understand it it is safe to proceed at all! This is forcing the fail fast upgrade treadmill that I fear will upset users and make parquet unsuitable for long term storage :( > > > p.s. I really like Will's formulation of the tension between long term > stable storage and platform vendor upgrade cycles. Thanks :blush: >
