etseidl commented on issue #563: URL: https://github.com/apache/parquet-format/issues/563#issuecomment-4625388069
I want to give some additional context in light of @rdblue's [comment](https://lists.apache.org/thread/xypskl7qow0xfot73pczfkbs2jy7vldg) on the voting thread: > This isn't introducing a beneficial feature like encryption or a new encoding. It is just omitting metadata as a minor cleanup While the change itself is small (changing a single word in the thrift file), the impact can be quite substantial. My group currently has Parquet files in use with a moderately sized but fairly deeply nested schema (23 leaf columns, max def level 11). One file with this schema has a footer that consumes 89103 bytes. Rewriting that file without `path_in_schema` the footer is 56841 bytes. So just for this modest example, 36% of the bytes in the footer are `path_in_schema`. I did a quick test of decoding that footer with the thrift generated C++ parser, and found that 20% of the time to decode was spent on `path_in_schema`. Granted, this is a small difference compared to the size of the file (128MB), but as we move towards row-based point access scenarios where decoding the metadata begins to dominate, this added overhead can become massive. For instance, an extreme case [reported](https://lists.apache.org/thread/0mp06g0r27s0ynsg3pk54zl5bqc249wg) on the M/L has a footer of 367 MB, wi th 60% of that taken up by `path_in_schema`. I don't think it's fair to characterize this issue as "minor". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
