gaborkaszab commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4452906774
########## format/spec.md: ########## @@ -1801,6 +1804,44 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`, each as a whole, including all of its nested fields. Each ID must identify a field in the schema of the file it modifies, which is the table schema for a data file and the tracked file schema for a leaf manifest, or a row lineage field required by [Writing Column Files](#writing-column-files). Fields nested within a list or map must not be listed in `field_ids`. + +#### Writing Column Files + +The rules below apply to column files of data files. Rules for writing column files of leaf manifests are not defined here. + +A column file stores the value of each of its listed fields for every row of the data file. Values that did not change are still included, and values for deleted rows should be null. + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The other fields of the MODIFIED entry must match the previous entry, except for the following: + +- `partition`: + - If an added column file lists a field that is, or contains, a source field of the data file's partition spec, every row that is not deleted must have the same partition values after the update, and `partition` must be set to those values. Review Comment: "every row that is not deleted must have the same partition values after the update" Is this a writer requirement? When running `UPDATE tbl SET col_a = fn(col_b)` and `col_a` is a partition column do we have to fail if we see that `fn(col_b)` returns values spanning multiple partitions? Or should the writer just reject updating partition columns with column files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
