alamb commented on PR #598: URL: https://github.com/apache/parquet-format/pull/598#issuecomment-4968380674
> 1. It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data. My understanding is that the second feature would be classified as "forward incompatible" and thus could not be written to a new file without changing the embedded version / feature listing > 2. It potentially breaks more readers then necessary. Consider a new format version that adds a compression algorithm. This new version is bumped to be the the default writer version, but the user chooses a different compression algorithm for there file. Using the new format version breaks old readers that don't understand it. This can be solved by having writers try to dynamically "downgrade" but it adds more complexity that writers need to manage. It duplicates the information already in the footer (i.e. the CompressionCodec enum). This is true -- on the other hand writers could reduce the impact by emitting the lowest version number required to read all features in the file (rather than the most recent feature level the writer supports). The feature set could be restricted as necessary. I agree this is more coarse grained than a feature-by-feature (or buckets of features) listing, but it is simpler to communicate and implement, which is why I think it is worthc onsidering -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
