anuragmantri commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4342954794
########## format/spec.md: ########## Review Comment: I would keep it as is because this section describes capabilities. Column files is a way to achieve that capability. ########## format/spec.md: ########## Review Comment: Added column files to that sentence. ########## format/spec.md: ########## @@ -155,6 +155,7 @@ Version 4 of the Iceberg spec adds support for relative locations in metadata, e * **Delete manifest** -- A file that lists delete files to be associated with data files at planning time. * **Data file** -- A file that contains rows of a table. * **Delete file** -- A file that encodes rows of a table that are deleted by position or data values. +* **Column file** -- (v4+) A file that stores the values of one or more fields for each row of a data file, or for each entry of a v4 leaf manifest. Review Comment: Agreed, the dense layout belongs in the section, not the definition. Updated to your wording. ########## format/spec.md: ########## Review Comment: Added a pointer from Row Lineage to the Column Files section instead of repeating the rules in two places. ########## format/spec.md: ########## Review Comment: Not discussed so far. `replace` doesn't fit, since a column update changes table data. `overwrite` is the closest of the current values. In any case I'd rather not put spec text on this until the v4 commit API takes shape, so I'll leave it open here. ########## format/spec.md: ########## @@ -1065,7 +1066,7 @@ The `file_sequence_number` field represents the sequence number of the snapshot When adding a new file to a leaf manifest, its data and file sequence numbers are set to `null` because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing `null` with the manifest's sequence number from the snapshot root file. It is also possible to add a new file with data that logically belongs to an older sequence number. In that case, the data sequence number must be provided explicitly and not inherited. However, the file sequence number must be always assigned when the snapshot is successfully committed. -The data sequence number is inherited only if the entry status is ADDED, or MODIFIED by adding a column file. The file sequence number is inherited only if the entry status is ADDED. When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time. +The data sequence number is inherited only if the entry status is ADDED, or MODIFIED by adding a column file. The file sequence number is inherited only if the entry status is ADDED. When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time, except that the data sequence number of a MODIFIED entry that adds a column file is the sequence number of the snapshot that adds it (see [Column Updates](#column-updates)). Review Comment: Agreed, removed. ########## format/spec.md: ########## Review Comment: Added a sentence to "Equality Delete Files" and "Adding Column Files" sections that eq deletes should be converted. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. Review Comment: Good catch, updated to use that wording. A stored null in the previous file means "inherit", so copying it would make unchanged rows look updated. `_last_updated_time` isn't in the spec yet, so we'll extend this rule when #18526 lands, or in whichever of the two PRs merges second. Thanks @stevenzwu. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: +1 I'll rely on #18585 for this. ########## format/spec.md: ########## @@ -1460,6 +1461,82 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Column updates allow modifying individual columns of existing data files without rewriting the entire file. Updated column values are stored in separate _column files_ that are associated with a base data file via the `column_files` field on the base file's entry. Column updates are added in v4 and are not supported in earlier format versions. + +A column file is a standard data file, written in the table's file format, that contains one or more updated or newly added columns plus a required `_pos` column, for a single base data file. Column files are valid only when the base entry's `content_type` is `0` (DATA). + +A column file entry tracks only the following: + +* `field_ids` -- the top-level field ids physically present in the column file (updated or newly added columns) +* `location` -- the path to the column file +* `file_size_in_bytes` -- the total size of the column file in bytes + +Column file entries do not carry their own statistics, record count, or sequence number. Statistics for updated fields are merged into the base entry's content stats (see [Write Rules](#write-rules)), and column files are tracked by a single `latest_column_file_snapshot_id` on the base entry's tracking information (see [Column File Tracking](#column-file-tracking)). + +#### Row Matching + +A column file maps its rows to the base data file through a required `_pos` column that records, for each row in the column file, the ordinal position of that row in the base data file. This positional mapping is the only relationship the spec defines between a column file and its base file; the physical representation of a column file is otherwise an implementation detail. + +In particular, the spec does not mandate whether a column file is row-aligned with the base file (one entry per base row, using filler values at deleted or unmodified positions) or contains only a subset of rows (for example, only the rows that are live after applying deletion vectors). A writer may choose either representation. Readers must therefore handle both directions defensively: + +* A `_pos` value present in the column file that is no longer a live row in the base data file (for example, the row was deleted by a deletion vector) must be ignored. +* A live base row position that has no corresponding entry in the column file must fall back to the base data file's value for that field. + +Because every live row's value for an updated field is resolvable from a single file, a reader never joins a column file with the base file to reconstruct a column: a projected field is read from exactly one file. + +#### Invariants + +The following invariants must hold for column files: + +* At most one column file entry per field id for a given base data file. If multiple columns are updated together, they may be stored in the same column file. +* For complex types (structs, maps, lists), the entire top-level field is replaced. The `field_ids` in a column file entry refer to top-level field ids. + +#### Write Rules + +Writers producing column updates must follow these rules: + +1. The writer produces a column file containing the updated field(s) for the affected base data file, together with the `_pos` column mapping each row to its position in the base data file. Whether the column file includes filler values for deleted or unmodified positions is an implementation choice (see [Row Matching](#row-matching)). Engines should use delta encoding for the `_pos` column to minimize storage overhead. +2. When the same field is updated again, the writer produces a new column file that carries over the field's previous values and replaces the existing column file entry for that field id, so that each field id appears in at most one column file entry. +3. When a column file is added or replaced, the writer must update the base entry's statistics (`lower_bounds`, `upper_bounds`, `null_value_counts`, `value_counts`, `nan_value_counts`) for the updated field ids only. Statistics for fields that are not present in the column file must not be changed. This keeps the base entry's content stats usable for file-level pruning during scan planning without opening column files. +4. Statistics need not be exact, but they must be self-consistent for each field. A column file may contain a different number of rows than the base data file (because of applied deletes or filler values), so the `value_counts` and `null_value_counts` for an updated field may differ from those of other fields in the same entry. Bounds must follow the usual requirements (the lower bound is less than or equal to all live values and the upper bound is greater than or equal to all live values) but need not be tight. Writers must not write a field's statistics in a way that is inconsistent with that field's live values. +5. On commit, the base entry is marked `REPLACED` (3) and a successor entry with the same data file location is written as `MODIFIED`, carrying the updated `column_files` list. The successor entry's tracking information records the updated row positions in the `replaced_positions` bitmap and sets `latest_column_file_snapshot_id` to the committing snapshot. The `REPLACED`/`MODIFIED` pair may be co-located in the same manifest or split across manifests; readers reconcile the pair by data file location. + +#### Read Rules + +Readers resolve projected columns as follows: + +1. For each projected field id, if a matching entry exists in `column_files`, the column is read from that column file. Otherwise, the column is read from the base data file. +2. Values are aligned to base file positions using the `_pos` column, handling missing and extra positions defensively as described in [Row Matching](#row-matching). Deletion vectors are applied to the result. +3. If a query projects only fields that are not present in any column file, column files are not opened. +4. If a query projects only fields that are entirely covered by column files, the base file's data columns for those fields can be skipped. + +#### Interaction with Deletion Vectors + +Deletion vectors and column files are independent overlays on a base data file. A deletion vector always wins: a row deleted by a deletion vector is never surfaced to readers, regardless of whether a column file contains a value for that row's position and regardless of commit order. A position that is absent from a column file is not, by itself, an indication that a row is deleted; deletion is determined solely by the deletion vector. + +#### Interaction with Equality Deletes + +Column updates and equality deletes are mutually exclusive for a given data file. Applying an equality delete after a column update would require evaluating the delete predicate against the original row state rather than the updated state, which the read path does not support. Writers are expected to convert equality deletes to deletion vectors before applying column updates, so that a data file carrying column updates has no associated equality deletes. + +#### Compaction + +When a `RewriteDataFiles` operation compacts a data file that has associated column files, the compacted output must merge the base file and all column files into a single new data file. The resulting entry has `column_files` set to null (no remaining column files). + +#### Column File Tracking + +Column files do not carry their own sequence number. Instead, the base data file's tracking information records a single `latest_column_file_snapshot_id`: the id of the snapshot that added the most recent column file for that base file, or null when the base file has no column files. Adding a column file bumps the base data file's data sequence number, and that repurposed sequence number drives row lineage and change detection: + +* Rows in a column file with a null `_last_updated_sequence_number` inherit the base data file's (bumped) data sequence number. This applies to rows whose values were updated in the commit that produced the column file. +* Rows that carry over unchanged values from a prior column file retain their original `_last_updated_sequence_number`, which the writer materializes as a physical column in the column file. +* The `_row_id` for updated rows is unchanged; it is derived from the base data file's `first_row_id` and the row's position. Column files do not reassign row ids. Review Comment: Small correction to my earlier reply: after review I dropped the list of fields a column file must not contain. Readers ignore fields that aren't in `field_ids`, and `field_ids` can't list `_row_id`, so a stray `_row_id` has no effect. Adding column files still doesn't change row IDs, so the regular Row Lineage rules apply. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. Review Comment: Agreed, removed. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files Review Comment: Added. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). Review Comment: Thanks, I restructured along those lines. Let me know what you think. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: You are right, a MODIFIED entry written to a new manifest gets aggregated with the rest. It isn't specific to column updates, so I'll look separately at whether the spec needs wording for it. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. Review Comment: Simplified to "If an updated field is, or contains, a field in the sort order, `sort_order_id` must be null." I kept "or contains" so a nested sort field is covered. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. Review Comment: Reworded: `column_files` contains the previous entry's column files and the new column files. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: Review Comment: Agreed. The tracking fields now point to Updating Tracking Metadata and Sequence Number Inheritance. I kept the explicit `snapshot_id` rule since Péter asked for it. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. Review Comment: Agreed, dropped the mention of deleted rows, and combined it with Péter's wording so unchanged rows get the non-null value a reader of the previous entry would see. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates Review Comment: Make sense. I combined them under Column Files section. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. Review Comment: Simplified to "`deletion_vector` must contain all existing deletes of the data file, including equality deletes." ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. + +Deletion vectors in delete manifests that reference the data file must be removed (see [Deletion Vectors](#deletion-vectors)). After column files are added, delete files in delete manifests do not apply to the data file (see [Scan Planning](#scan-planning)). Review Comment: Agreed, it follows from the bump. Removed the sentence and put the explanation in Equality Delete Files. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). Review Comment: Done. The file-level rule is now just "the same row count". Position alignment moved to Reading Column Files, and the deleted-row nulls moved to Adding Column Files, merged with the optional-field rule. ########## format/spec.md: ########## Review Comment: Good catch. I updated the wording to talk about live entries, and noted that a REPLACED entry and the MODIFIED entry that replaces it don't violate the rule. ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning -Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. +Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. Review Comment: Agreed, removed. A column update that touches a partition source field now always sets `spec_id` and `partition` to null, so the invariant holds without the extra sentence. @stevenzwu on defining "data file" as the base file plus column files: I'd rather not. Terms, `location` and `record_count` all mean the base file, so it would conflict with them. ########## format/spec.md: ########## @@ -1291,6 +1292,8 @@ When expiring snapshots, retention policies in table and snapshot references are A file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that the file is not live in any remaining snapshot [1]. +A [column file](#column-updates) is live in a snapshot if a live entry in that snapshot has it in `column_files`. A column file may be deleted from the file system when it is not live in any remaining snapshot. Review Comment: Removed. One case to keep in mind: a column file superseded by a later update while its data file is still live has no DELETED entry of its own. I'm leaving that to implementations for now, but happy to add wording if it needs it. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. Review Comment: Agreed, updated: `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. Review Comment: Good question, and I don't think it's covered yet. Merging an equality delete into the deletion vector makes those rows look newly deleted if a reader diffs the REPLACED and MODIFIED entries' deletion vectors, even though they were deleted in an earlier snapshot. A change reader can get the previously deleted set from the REPLACED entry: its deletion vector plus the delete files that applied to it. I agree with @stevenzwu here on keeping the spec to writer behavior here for now. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. Review Comment: Updated to your wording. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. Review Comment: Good point. Changed to "a top-level field must be optional in the column file's schema if the column file stores null for it", and merged it with the deleted-row rule so the reason is stated next to it. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. Review Comment: I made a change and it now says. A column update that touches a partition source field now always sets `spec_id` and `partition` to null. One case that came to my mind is tables migrated by metadata only (from Hive to Iceberg for example). Those data files don't contain the partition columns, and Column Projection fills them from the `partition` struct. Nulling the tuple would make columns like `dt` read as null. Options: 1. require any missing identity source columns to be written into a column file before the tuple is nulled, 2. forbid updating partition sources on files that depend on the tuple. Thoughts? ########## format/spec.md: ########## @@ -1221,7 +1222,7 @@ For example, an `events` table with a timestamp column named `ts` that is partit The inclusive projection for an unknown partition transform is _true_ because the partition field is ignored and not used in filtering. -Data files that match the query filter must be read by the scan. +Data files that match the query filter must be read by the scan. Data files with column files must be read as described in [Column Updates](#column-updates). Review Comment: Agreed, removed the Scan Planning sentence. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. Review Comment: Thanks, I took your wording and added a link to `column_files`. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. Review Comment: Trimmed it to a short list of the three terms the rules depend on (previous entry, updated field, updated row) and dropped "MODIFIED entry". ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning Review Comment: Added this to `Adding column files` section. Let me know if you agree. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
