emkornfield commented on code in PR #542:
URL: https://github.com/apache/parquet-format/pull/542#discussion_r2636149126
##########
src/main/thrift/parquet.thrift:
##########
@@ -958,6 +958,24 @@ union ColumnCryptoMetaData {
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
+ *
+ * As of December 2025, the only known use-case for this field is writing
summary
+ * parquet files (i.e. "_metadata" files). These files consolidate footers
from
+ * multiple parquet files to allow for efficient reading of footers to
avoid file
+ * listing costs and prune out files that do not need to be read based on
statistics.
+ * This is legacy feature as modern table formats (e.g. Iceberg, Hudi and
Delta Lake)
+ * are more scalable and serve effectively the same purpose.
+ *
+ * There is no other known use-case for this field. Specifically, there are
no known
+ * readers that will read externally stored column data if this field is
populated
+ * within a standard parquet file.
+ *
+ * Writers should not populate this field except for in parquet summary
files. Readers
+ * should ensure this field is empty.
Review Comment:
Not sure if this was also referring to below as well. But I updated some
language above this specific sentence in the paragraph above to also clarify
that using this for reading external columns is not considered part of the
specification (which is reflected by current implementation status).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
