stefan-grafberger commented on code in PR #635: URL: https://github.com/apache/parquet-format/pull/635#discussion_r4458641783
########## LogicalTypes.md: ########## @@ -712,8 +713,9 @@ If set and non-zero, readers must seek to this offset to retrieve the referenced The byte length of the referenced data. Must be zero or a positive integer if set; a value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), -in which case the range runs to the end of the referenced file. +It may be omitted for a whole-file external reference (`uri` set, `offset` not set), +in which case the range runs to the end of the referenced file. `size` may also be set +with `inline` and without `uri`, in which case it must equal the length of `inline`. Review Comment: Good point, I fully understand the hesitation to write it twice. We also had internal discussions about this but believe it's worthwhile to have the option to store the inline size separately. Besides pruning, the other big benefit is projections: a query that needs the size of each FILE value, e.g. to list some file identifiers and the file sizes from a FILE column that mixes inline values and external references, can be answered from a few small columns. Without a separate size, the reader has to read the potentially very large pages of the inline subcolumn to get the lengths of inline values. That gets even more expensive if the inline field is stored in separate Parquet files or in non-contiguous pages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
