gaborkaszab commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4323455619
########## format/spec.md: ########## Review Comment: Since we mention V4 and colocated deletion vectors already, shouldn't we also mention column files too? ########## format/spec.md: ########## Review Comment: This seems no longer true: We can multiple produce entries for the same file in a snapshot when it's REPLACED + MODIFIED. Not tightly related to this PR though, but we could discuss including it here or opening a separate PR. ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning -Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. +Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. Review Comment: Maybe it's just me, but I don't get the point of this sentence: `For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files.` Do you say that the invariant of `all records in a data file should have the same partition values` is not true when we have column files on the data file? ########## format/spec.md: ########## Review Comment: More general `column files` instead of `column updates`? ########## format/spec.md: ########## Review Comment: This section explains what is expected when we write an entire row into a new file. Shouldn't we also cover the column update case here, mentioning that `_row_id` is not expected in the column file, but `_lusn` is a must? I see a section below that covers this in the context of writing column files, but I believe that the "Row Lineage" section is more self-contained with covering column updates too. ########## format/spec.md: ########## Review Comment: Now that we include constraints in V4 for eq-deletes, shouldn't we also include that writers must rewrite existing eq-deleted rows of data file into DVs when committing column updates to those data files? Also another constraint we introduce is that we must bump the data seq num of such data files to avoid applying the existing eq-deletes on these files. Probably worth mentioning here, but on the "how to write column updates" section too. ########## format/spec.md: ########## @@ -553,7 +554,7 @@ Note that: ### Partitioning Review Comment: I recall, we talked about setting the partition to "unpartitioned" when updating partition fields using column files. I'm not sure how much we agreed on that, but would be nice to clarify in this section, what happens to partitioning in the context of column updates on partition columns. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). Review Comment: I've read these 2 paragraphs and for me it doesn't read that naturally. Maybe as an alternative: ``` A column file is positionally aligned with its data file and must have the same row count. Row i of the column file corresponds to row i of the data file. A column file stores values for the fields identified by its field_ids for the rows in its data file. Each ID in field_ids must identify a top-level field in the table schema used to write the column file. The field_ids of a column file might or might not overlap with the field IDs in the data file. A column file must also contain the _last_updated_sequence_number row lineage field, written as described in #adding-column-files. ``` Sorry, haven't formatted this to fit into the spec, focused on the content. WDYT? Update: I'm somewhat hesitant with the `_lusn` field being a must. That constraint would probably break with column families, and maybe also break if we decide to write multiple column files with a single commit, because then only one of them must contain that field. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. Review Comment: What about aggregated stats on the manifest level? Should we re-aggregate them when updating a column or should we invalidate them? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. Review Comment: Not sure about "must". For column families required field from the schema could still be required in the column file. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files Review Comment: Maybe add that readers expected to stitch the different column files and base file together first, and then apply the delete filter after? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. Review Comment: I find these explanations with "previous entry" and such a bit hard to read. Can't just simply say `If we update one or more field that are part of the sort order fields, then sort_order_id must be set to null` ? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. + +Deletion vectors in delete manifests that reference the data file must be removed (see [Deletion Vectors](#deletion-vectors)). After column files are added, delete files in delete manifests do not apply to the data file (see [Scan Planning](#scan-planning)). Review Comment: Is this true? Don't we guarantee this by bumping the data sequence number? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. + - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. + - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. Review Comment: "for every other row that is not deleted." I wouldn't mention deleted rows here, we don't really have ay assumptions to their `_LUSN` value. Maybe a simplified version: `Its value must be null for each updated row, and for the non-updated rows it must be the _last_updated_sequence_number read from the previous entry` ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. +- `column_files`: + - New column files are added to the previous entry's `column_files`. Review Comment: This sentence seems misleading, gives me the impression that we add the new column files into the previous entry, but we rather add them to the new entry. ########## format/spec.md: ########## @@ -1065,7 +1066,7 @@ The `file_sequence_number` field represents the sequence number of the snapshot When adding a new file to a leaf manifest, its data and file sequence numbers are set to `null` because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing `null` with the manifest's sequence number from the snapshot root file. It is also possible to add a new file with data that logically belongs to an older sequence number. In that case, the data sequence number must be provided explicitly and not inherited. However, the file sequence number must be always assigned when the snapshot is successfully committed. -The data sequence number is inherited only if the entry status is ADDED, or MODIFIED by adding a column file. The file sequence number is inherited only if the entry status is ADDED. When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time. +The data sequence number is inherited only if the entry status is ADDED, or MODIFIED by adding a column file. The file sequence number is inherited only if the entry status is ADDED. When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time, except that the data sequence number of a MODIFIED entry that adds a column file is the sequence number of the snapshot that adds it (see [Column Updates](#column-updates)). Review Comment: `, except that the data sequence number of a MODIFIED entry that adds a column file is the sequence number of the snapshot that adds it` When adding a column file the MODIFIED entry inherits the data sequence number. This is covered by the first sentence of this paragraph, I don't think we should extend this further with explaining from a different angle. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. Review Comment: Didn't we say once that when we update a partition column with a column update, then we set the partition to unpartitioned? I might remember wrong, though. ########## format/spec.md: ########## Review Comment: Reading this part of the spec reminds me: Have discussed about the operation type when doing column updates? Is that an `overwrite`? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. Review Comment: I'm not sure we should mention what the column file must not contain. Rather covering what is expected to be in the column file seems enough. Anyway, `_row_id` and `_pos` are technically feasible in the column file, it would be "garbage", because we just won't use them. I'd just drop this sentence. ########## format/spec.md: ########## @@ -155,6 +155,7 @@ Version 4 of the Iceberg spec adds support for relative locations in metadata, e * **Delete manifest** -- A file that lists delete files to be associated with data files at planning time. * **Data file** -- A file that contains rows of a table. * **Delete file** -- A file that encodes rows of a table that are deleted by position or data values. +* **Column file** -- (v4+) A file that stores the values of one or more fields for each row of a data file, or for each entry of a v4 leaf manifest. Review Comment: nit: `for each row of` and `for each entry of` seems an implementation detail that we decided on the dense representation. Might not be relevant here when we define column files. Suggestion: `A file that stores the values of one or more fields of a data file, or a v4 leaf manifest` ? ########## format/spec.md: ########## @@ -1291,6 +1292,8 @@ When expiring snapshots, retention policies in table and snapshot references are A file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that the file is not live in any remaining snapshot [1]. +A [column file](#column-updates) is live in a snapshot if a live entry in that snapshot has it in `column_files`. A column file may be deleted from the file system when it is not live in any remaining snapshot. Review Comment: A column file is tied to a data file. If we garbage collect the data file, we should garbage collect its column files too. I don't think we should introduce the concept of live or not live column files to the spec. With column families, this might change, but let's not design that at this point. ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: Review Comment: Isn't requirements for the mODIFIED entry's tracking already detailed somewhere else in the spec that we can simply refer to and not duplicate? ########## format/spec.md: ########## @@ -1653,6 +1656,61 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Updates + +Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. + +In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. + +#### Column Files + +A column file is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. + +A column file stores the entire value of each field whose ID is in its `field_ids`. Each ID in `field_ids` must be the ID of a top-level field in the table schema used to write the column file, or the ID of a row lineage field required by [Adding Column Files](#adding-column-files). + +A column file must have the same row count as its data file, and row `i` of the column file stores values for row `i` of the data file. Each row stores the value of each listed field after the column update, except that values in rows deleted by the MODIFIED entry's `deletion_vector` should be null. Row lineage fields are written as described in [Adding Column Files](#adding-column-files). + +Column file requirements: + +- Each top-level field must be optional in the column file's schema, even if the field is required in the table schema. +- For a top-level field that is required in the table schema, a non-null value must be stored in every row that is not deleted by the MODIFIED entry's `deletion_vector`. +- A column file must not contain the [reserved](#reserved-field-ids) `_row_id` or `_pos` fields. + +#### Reading Column Files + +A projected field whose top-level field ID is in a column file's `field_ids`, including `_last_updated_sequence_number`, must be read from that column file. All other projected fields are read from the data file using [Column Projection](#column-projection). + +Readers must ignore top-level fields in a column file whose IDs are not in its `field_ids`. A null value in a column file does not indicate that a row is deleted. + +#### Adding Column Files + +When column files are added to a data file, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The MODIFIED entry must have the previous metadata of the data file, except for the following: + +- `tracking`: + - `tracking.status` is MODIFIED. + - `tracking.snapshot_id` must not be null. + - `tracking.modified_snapshot_id` must record the snapshot ID in which the column files were added. + - `tracking.sequence_number` is the sequence number of the snapshot in which the column files were added. See [Sequence Number Inheritance](#sequence-number-inheritance). +- `spec_id` and `partition`: + - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. + - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. +- `sort_order_id`: + - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. +- `deletion_vector`: + - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. Review Comment: Can't we rephrase this to be simpler, like `existing deletes on the data file should be written into deletion_vector including equality deletes` ? Or something similar :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
