anuragmantri commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4510227218
########## format/spec.md: ########## @@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`, each as a whole, including all of its nested fields. Each ID must identify a field in the schema of the file it modifies, which is the table schema for a data file and the tracked file schema for a leaf manifest, or a row lineage field required by [Writing Column Files](#writing-column-files). Fields nested within a list or map must not be listed in `field_ids`. Review Comment: Thanks, I took these suggestions. Per @huaxingao's suggestion, the overlap rule is moved to `column_files` requirements. I also added one sentence to the read rule: "it applies to each field nested in a projected struct". Without it, projecting `customer` when a column file lists `address` would read the whole struct, including the old `address`, from the data file. ########## format/spec.md: ########## @@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`, each as a whole, including all of its nested fields. Each ID must identify a field in the schema of the file it modifies, which is the table schema for a data file and the tracked file schema for a leaf manifest, or a row lineage field required by [Writing Column Files](#writing-column-files). Fields nested within a list or map must not be listed in `field_ids`. Review Comment: Good catch. The rule now applies when the column file is added, and I added "IDs that aren't projected are ignored" in the "Reading" section. Field IDs are never reused, so this should be safe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
