pvary commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4324960384
########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. Review Comment: Is it discussed how a change data reader will detect that these rows were deleted previously, and they are not new deletes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
