emkornfield commented on PR #598: URL: https://github.com/apache/parquet-format/pull/598#issuecomment-4959181005
> Thanks @emkornfield -- the only thing I see missing here is how the parquet writer would communicate to readers what features were used in a partcular file. @alamb I'm specifically https://github.com/apache/parquet-format/pull/588 (PARX magic number is roughly what I was thinking here) but I specifically removed it from this PR because I think we are trying to build consensus on a "logical level/process" first, because getting to the physical level. > To make this proposal an improvement over the status quo, I think we would also need to add an extra field to the footer metadata that encodes the minimum parquet-format version of forward incompatible features used (aka the minimium parquet-format version support required to successfully read the file) I'm hesitant to use version number as something implementations use to consume to determine if they could read the file (I do think it is very useful debug information and can help provide useful errors to end users,). I think it fails in two directions: 1. It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data. 2. It potentially breaks more readers then necessary. Consider a new format version that adds a compression algorithm. This new version is bumped to be the the default writer version, but the user chooses a different compression algorithm for there file. Using the new format version breaks old readers that don't understand it. This can be solved by having writers try to dynamically "downgrade" but it adds more complexity that writers need to manage. It duplicates the information already in the footer (i.e. the CompressionCodec enum). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
