stevenzwu commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4517843700
########## format/spec.md: ########## @@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`. A listed struct includes all fields nested in it. In this section, a struct _contains_ a field if the field is nested in the struct at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of the file it modifies, or a row lineage field required by [Writing Column Files](#writing-column-files). The schema of a data file is the table schema. The schema of a leaf manifest is the tracked file schema. Fields nested in a list or map must not be listed in `field_ids`. + +#### Writing Column Files + +A column file stores the value of each of its listed fields for every row of the file it modifies. Values that did not change are still included, and values for deleted rows should be null. + +The added column files must list each field that the update changes, or a struct that contains that field. If an update changes whether a struct is null, the update changes that struct. + +When column files are added to a data file or a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. For a leaf manifest, `tracking.sequence_number` must not change. The fields of the MODIFIED entry outside `tracking` must match the previous entry, except for the following: + +- `partition`: + - For a data file, an added column file may list a field that is, or contains, a source field of the data file's partition spec only if every row that is not deleted has the same partition values after the update. `partition` must be set to those values. +- `sort_order_id`: + - If an added column file lists a field that is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for a field listed in an added column file, and for its nested fields, must be computed from that column file, or omitted. + - For a leaf manifest, stats that summarize a listed field must be recomputed or omitted. +- `tight_stats`: + - `tight_stats` may be true only if `content_stats` are tight for the rows that are live after the update (see [Stats Tightness](#stats-tightness)). +- `deletion_vector`: + - For a data file, every delete that applied to it before the column files were added must be written into the MODIFIED entry's `deletion_vector`. +- `column_files`: + - Each field listed in an added column file must be removed from the `field_ids` of the previous entry's column files. Each field that a listed struct contains must also be removed. A column file with no remaining `field_ids` must be removed. + - For a data file, one of the added column files must take responsibility for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing the data file or previous column file that stored it. In that column file: + - If the added column files change the row, `_last_updated_sequence_number` must be set to `null` (so that the sequence number of the snapshot that added the column files replaces the current value). + - If the added column files do not change the row and the row is not deleted, the existing `_last_updated_sequence_number` value, with null resolved as in [Row lineage assignment](#row-lineage-assignment), must be copied to the new column file. + +#### Reading Column Files + +A projected field must be read from the column file that lists the field or a struct that contains it. This applies to each field nested in a projected struct. All other projected fields are read from the file it modifies, using [Column Projection](#column-projection) for a data file. IDs in `field_ids` that are not projected are ignored. + +If a struct is null in a row, each field that the struct contains is null in that row. This applies when the struct and the nested field are read from different files. Review Comment: "This applies when" reads as "only when", and the sentence doesn't say that a column file's value is ignored: ```suggestion If a struct is null in a row, every field it contains is null in that row, including a nested field read from a column file. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
