anuragmantri commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4623191101
########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. +- Each field whose value changes must be in `field_ids`. +- Each struct that changes between null and non-null must be in `field_ids`. +- If a struct changes from null to non-null, each field directly nested in it must be in `field_ids`, even if the field is null. + +See [Column file example](#column-file-example). + +When column files are added to a data file or a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry outside `tracking` must match the previous entry, except for the following: + +- `partition`: + - For a data file, an added column file's `field_ids` may have the ID of a partition source field, or of a field that contains one, only if all live rows share the same partition values after the update. `partition` must be set to those values. +- `sort_order_id`: + - If an added column file lists a field that is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for a field listed in an added column file, and for its nested fields, must be recomputed or omitted. + - For a leaf manifest, stats that summarize a listed field must be recomputed or omitted. +- `tight_stats`: + - If `tight_stats` is true, `content_stats` must be tight for the live rows after the update (see [Stats Tightness](#stats-tightness)). +- `deletion_vector`: + - For a data file, every delete that applied to it before the column files were added must be written into the MODIFIED entry's `deletion_vector`. +- `column_files`: + - Each field listed in an added column file must be removed from the `field_ids` of the previous entry's column files. A column file with no remaining `field_ids` must be removed. + - For a data file, one of the added column files must have the `_last_updated_sequence_number` ID in its `field_ids`, so that rows changed by the added column files get a new `_last_updated_sequence_number`. In that column file: + - If the added column files change the row, `_last_updated_sequence_number` must be set to `null` (so that the sequence number of the snapshot that added the column files replaces the current value). + - If the added column files do not change the row and the row is not deleted, the existing `_last_updated_sequence_number` value, with null resolved as in [Row lineage assignment](#row-lineage-assignment), must be copied to the new column file. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. Review Comment: Done, with "fields in" so nested fields are included. I also added "Fields nested in a list or map are not valid." here. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. +- Each field whose value changes must be in `field_ids`. +- Each struct that changes between null and non-null must be in `field_ids`. +- If a struct changes from null to non-null, each field directly nested in it must be in `field_ids`, even if the field is null. + +See [Column file example](#column-file-example). + +When column files are added to a data file or a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry outside `tracking` must match the previous entry, except for the following: + +- `partition`: + - For a data file, an added column file's `field_ids` may have the ID of a partition source field, or of a field that contains one, only if all live rows share the same partition values after the update. `partition` must be set to those values. +- `sort_order_id`: + - If an added column file lists a field that is, or contains, a field in the sort order, `sort_order_id` must be null. +- `content_stats`: + - Stats for a field listed in an added column file, and for its nested fields, must be recomputed or omitted. + - For a leaf manifest, stats that summarize a listed field must be recomputed or omitted. +- `tight_stats`: + - If `tight_stats` is true, `content_stats` must be tight for the live rows after the update (see [Stats Tightness](#stats-tightness)). +- `deletion_vector`: + - For a data file, every delete that applied to it before the column files were added must be written into the MODIFIED entry's `deletion_vector`. +- `column_files`: + - Each field listed in an added column file must be removed from the `field_ids` of the previous entry's column files. A column file with no remaining `field_ids` must be removed. + - For a data file, one of the added column files must have the `_last_updated_sequence_number` ID in its `field_ids`, so that rows changed by the added column files get a new `_last_updated_sequence_number`. In that column file: + - If the added column files change the row, `_last_updated_sequence_number` must be set to `null` (so that the sequence number of the snapshot that added the column files replaces the current value). + - If the added column files do not change the row and the row is not deleted, the existing `_last_updated_sequence_number` value, with null resolved as in [Row lineage assignment](#row-lineage-assignment), must be copied to the new column file. + +#### Reading Column Files + +Each projected field, including each field nested in a projected field, must be read from the column file whose `field_ids` has its ID. A field nested in a list or map is read with that list or map. Other projected fields are read from the base file, using [Column Projection](#column-projection) for a data file. A column file of a data file is also read using [Column Projection](#column-projection). + +If a struct is null in a row, every field nested in it is null in that row, including a field read from a column file. + +A column file and its base file are aligned by row position before the attached deletion vector is applied. The scan schema must be used for field types and nullability. Null values in a column file may represent deleted rows. + +#### Column file example + +This example demonstrates which field IDs a column file must have in its `field_ids`, using this schema and one row of a data file. Column files are named `column_file_1`, `column_file_2`, and so on. The `_last_updated_sequence_number` ID that each update also adds to `field_ids` is left out. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. Review Comment: Done. We still need this rule some I moved it to the paragraph on valid IDs. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. +- Each field whose value changes must be in `field_ids`. Review Comment: Done, thanks. ########## format/spec.md: ########## Review Comment: Added as note 2. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. +- Each field whose value changes must be in `field_ids`. +- Each struct that changes between null and non-null must be in `field_ids`. +- If a struct changes from null to non-null, each field directly nested in it must be in `field_ids`, even if the field is null. + +See [Column file example](#column-file-example). + +When column files are added to a data file or a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -2275,6 +2351,10 @@ Constraints are added in v4: * Snapshots may contain `constraint-statuses` * Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced, unless a field cannot be used in a `primary-key` constraint +Column files are added in v4. + +* Data files and v4 leaf manifests may have column files that store the values of one or more fields (see [Column Files](#column-files)) Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -157,6 +157,7 @@ Version 4 of the Iceberg spec adds support for relative locations in metadata, e * **Delete manifest** -- A file that lists delete files to be associated with data files at planning time. * **Data file** -- A file that contains rows of a table. * **Delete file** -- A file that encodes rows of a table that are deleted by position or data values. +* **Column file** -- (v4+) A file that stores the values of one or more fields for a data file or a leaf manifest. Review Comment: Done, thanks. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. + +- A field nested in a list or map must not be in `field_ids`. When such a field changes, the list or map must be in `field_ids`. +- Each field whose value changes must be in `field_ids`. +- Each struct that changes between null and non-null must be in `field_ids`. +- If a struct changes from null to non-null, each field directly nested in it must be in `field_ids`, even if the field is null. + +See [Column file example](#column-file-example). + +When column files are added to a data file or a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry as described in [Updating Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the REPLACED entry may be written to different manifests. + +The `tracking` fields of the MODIFIED entry are set as described in [Updating Tracking Metadata](#updating-tracking-metadata) and [Sequence Number Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry outside `tracking` must match the previous entry, except for the following: + +- `partition`: + - For a data file, an added column file's `field_ids` may have the ID of a partition source field, or of a field that contains one, only if all live rows share the same partition values after the update. `partition` must be set to those values. Review Comment: Good catch. The partition bullet used to say the data file's partition spec and lost it in a rewording. Both bullets now name the data file's own partition spec and sort order. ########## format/spec.md: ########## @@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. Readers must ignore fields in a column file whose IDs are not in its `field_ids`. In this section, a field _contains_ every field nested in it, at any depth. + +When a column file is added, each ID in its `field_ids` must identify a field in the schema of its base file, or `_last_updated_sequence_number` (see [Writing Column Files](#writing-column-files)). The schema of a data file is the current table schema. The schema of a leaf manifest is the tracked file schema. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that contains the field is non-null after the update. Review Comment: What a reader returns, so a value stored under a null struct reads as null. Base file values under a null parent don't come back. If an update makes the parent non-null, each field directly nested in it must be in `field_ids`, so the column file replaces them. For example, if `customer` is null and an update sets `address`, then `customer`, `name` and `address` all go in `field_ids`, and the old `name` in the base file is never read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
