gaborkaszab commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4642826482


##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.

Review Comment:
   "An added column file must follow these rules for `field_ids`."
   Does this refer to the paragraphs after "Column Files" but before "Writing 
Column Files" ?
   I wonder what extra information this sentence gives. We defined the 
`field_ids` rules above, do we have to call out here that we have to obey those 
rules on the write path?



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.

Review Comment:
   "Fields nested in the struct are stored only if their IDs are also in 
`field_ids`."
   
   Does this mean that when we update an entire structure column, we have to 
list all the nested fields (including recursively fields from nested structs) 
in `field_ids`?



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+    - For a data file, an added column file's `field_ids` may have the ID of a 
source field of the data file's partition spec, or of a struct that one is 
nested in, only if all live rows share the same partition values after the 
update. `partition` must be set to those values.
+- `sort_order_id`:
+    - For a data file, if an added column file lists a field in the data 
file's sort order, or a struct that one is nested in, `sort_order_id` must be 
null.
+- `content_stats`:
+    - Stats for a field listed in an added column file, and for its nested 
fields, must be recomputed or omitted.
+    - For a leaf manifest, stats that summarize a listed field must be 
recomputed or omitted.
+- `tight_stats`:
+    - If `tight_stats` is true, `content_stats` must be tight for the live 
rows after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+    - For a data file, every delete that applied to it before the column files 
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+    - Each field listed in an added column file must be removed from the 
`field_ids` of the previous entry's column files. A column file with no 
remaining `field_ids` must be removed.
+    - For a data file, one of the added column files must have the 
`_last_updated_sequence_number` ID in its `field_ids`, so that rows changed by 
the added column files get a new `_last_updated_sequence_number`. In that 
column file:

Review Comment:
   ", so that rows changed by the added column files get a new 
`_last_updated_sequence_number`"
   
   I think we describe the "what" here and probably not the "why".  Maybe 
remove this part of the sentence?



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.

Review Comment:
   "fields in the `tracked_file` struct are valid"
   
   Can you add a link to `tracked_file` here?



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.

Review Comment:
   "The REPLACED entry must be a full entry in the root manifest."
   I'm not sure I understand what "full entry" means.



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+    - For a data file, an added column file's `field_ids` may have the ID of a 
source field of the data file's partition spec, or of a struct that one is 
nested in, only if all live rows share the same partition values after the 
update. `partition` must be set to those values.
+- `sort_order_id`:
+    - For a data file, if an added column file lists a field in the data 
file's sort order, or a struct that one is nested in, `sort_order_id` must be 
null.
+- `content_stats`:
+    - Stats for a field listed in an added column file, and for its nested 
fields, must be recomputed or omitted.
+    - For a leaf manifest, stats that summarize a listed field must be 
recomputed or omitted.
+- `tight_stats`:
+    - If `tight_stats` is true, `content_stats` must be tight for the live 
rows after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+    - For a data file, every delete that applied to it before the column files 
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+    - Each field listed in an added column file must be removed from the 
`field_ids` of the previous entry's column files. A column file with no 
remaining `field_ids` must be removed.
+    - For a data file, one of the added column files must have the 
`_last_updated_sequence_number` ID in its `field_ids`, so that rows changed by 
the added column files get a new `_last_updated_sequence_number`. In that 
column file:
+        - If the added column files change the row, 
`_last_updated_sequence_number` must be set to `null` (so that the sequence 
number of the snapshot that added the column files replaces the current value).
+        - If the added column files do not change the row and the row is not 
deleted, `_last_updated_sequence_number` must be copied to the new column file. 
A null value must be replaced with the data sequence number a reader of the 
previous entry would assign.
+
+#### Reading Column Files
+
+Each projected field, including each field nested in a projected field, must 
be read from the column file whose `field_ids` has its ID. A field nested in a 
list or map is read with that list or map. Other projected fields are read from 
the base file, using [Column Projection](#column-projection) for a data file. A 
column file of a data file is also read using [Column 
Projection](#column-projection).
+
+If a struct is null in a row, every field nested in it is null in that row, 
including a field read from a column file.
+
+A column file and its base file are aligned by row position before the 
attached deletion vector is applied. The scan schema must be used for field 
types and nullability. Null values in a column file may represent deleted rows.
+
+#### Column file example
+
+This example demonstrates which field IDs a column file must have in its 
`field_ids`, using this schema and one row of a data file. Unless a row says 
otherwise, that row in the base file is `customer = {name: …, address: {street: 
'1 Elm', city: 'Austin'}}`. Column files are named `column_file_1`, 
`column_file_2`, and so on. The `_last_updated_sequence_number` ID that each 
update also adds to `field_ids` is left out.
+
+```
+2: optional struct customer {
+  3: optional string name;
+  4: optional struct address {
+    5: optional string street;
+    6: optional string city;
+  }
+}
+8: optional list<string> tags; // element field ID 9
+```
+
+| Row before the update | Update | Valid column files | Invalid column files | 
Notes |
+|---|---|---|---|---|
+| `customer` is null | `SET name = 'andrew'` | `column_file_1` with `field_ids 
= [2, 3, 4]`, and `address` null | `column_file_1` with `field_ids = [3]` | 
`customer` must be in `field_ids`, or it is read as null from the base file and 
hides `name`. `address` is in `field_ids` because it is directly nested in 
`customer`, which changed from null to non-null. Its fields are not, because 
`address` is null after the update. |
+| `address` is `{street: '1 Elm', city: 'Austin'}` | `SET address = null` | 
`column_file_1` with `field_ids = [4]` | | `street` and `city` are not in 
`field_ids` because `address` is null after the update. |
+| `address` is null in `column_file_1` with `field_ids = [4]` | `SET street = 
'2 Oak'` | `column_file_2` with `field_ids = [4, 5, 6]`, and `city` null | 
`column_file_2` with `field_ids = [4, 5]` | `city` must be in `field_ids`, or 
it is read as `'Austin'` from the base file. `column_file_1` has no fields left 
and is removed. |

Review Comment:
   I'm wondering how a writer should know if the desired column file schema is 
[4, 5, 6] because there was a row where 4 was null, or [5] because all of the 
rows have non-null 4.
   
   This affects the set of columns we have to read as the input for the write 
and also affects the schema of the column file. Should we rely on null stats 
here? Or should we assume the worst and when updating a field of a struct we 
read all the members of the struct and decide the schema of the update file 
later?
   
   This might be especially interesting when updating some stats of a leaf 
manifest for a single field. Then if we are shooting for the worst case, we 
have to read the entire `content_stats` field, that is almost the entire leaf 
manifest.



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+    - For a data file, an added column file's `field_ids` may have the ID of a 
source field of the data file's partition spec, or of a struct that one is 
nested in, only if all live rows share the same partition values after the 
update. `partition` must be set to those values.
+- `sort_order_id`:
+    - For a data file, if an added column file lists a field in the data 
file's sort order, or a struct that one is nested in, `sort_order_id` must be 
null.
+- `content_stats`:
+    - Stats for a field listed in an added column file, and for its nested 
fields, must be recomputed or omitted.
+    - For a leaf manifest, stats that summarize a listed field must be 
recomputed or omitted.
+- `tight_stats`:
+    - If `tight_stats` is true, `content_stats` must be tight for the live 
rows after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+    - For a data file, every delete that applied to it before the column files 
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+    - Each field listed in an added column file must be removed from the 
`field_ids` of the previous entry's column files. A column file with no 
remaining `field_ids` must be removed.

Review Comment:
   "previous entry's column files"
   This might give an impression that we have to modify the previous, aka 
REPLACED entry.
   Maybe "must be removed from the `field_ids` of the existing column files"?
   We already mention added column files, probably we can say existing column 
files too.



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+    - For a data file, an added column file's `field_ids` may have the ID of a 
source field of the data file's partition spec, or of a struct that one is 
nested in, only if all live rows share the same partition values after the 
update. `partition` must be set to those values.
+- `sort_order_id`:
+    - For a data file, if an added column file lists a field in the data 
file's sort order, or a struct that one is nested in, `sort_order_id` must be 
null.
+- `content_stats`:
+    - Stats for a field listed in an added column file, and for its nested 
fields, must be recomputed or omitted.
+    - For a leaf manifest, stats that summarize a listed field must be 
recomputed or omitted.
+- `tight_stats`:
+    - If `tight_stats` is true, `content_stats` must be tight for the live 
rows after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+    - For a data file, every delete that applied to it before the column files 
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+    - Each field listed in an added column file must be removed from the 
`field_ids` of the previous entry's column files. A column file with no 
remaining `field_ids` must be removed.
+    - For a data file, one of the added column files must have the 
`_last_updated_sequence_number` ID in its `field_ids`, so that rows changed by 
the added column files get a new `_last_updated_sequence_number`. In that 
column file:
+        - If the added column files change the row, 
`_last_updated_sequence_number` must be set to `null` (so that the sequence 
number of the snapshot that added the column files replaces the current value).
+        - If the added column files do not change the row and the row is not 
deleted, `_last_updated_sequence_number` must be copied to the new column file. 
A null value must be replaced with the data sequence number a reader of the 
previous entry would assign.

Review Comment:
   "`_last_updated_sequence_number` must be copied to the new column file. A 
null value must be replaced with the data sequence number a reader of the 
previous entry would assign."
   
   Can we simplify this? Maybe:
   
   "`last_updated_sequence_number` is the `last_updated_sequence_number` of the 
row from the base file, either directly copied if non-null, or inherited from 
the previous entry if null."



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.

Review Comment:
   "if a rule requires it"
   It might not be trivial what rules we mean here. Can we be explicit when we 
introduce those rules, that "Here are the rules for field_ids" ?



##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.

Review Comment:
   Isn't there quite a bit of redundancy between these rules and the ones at 
line1812? The list, map and structure part seems to overlap with this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to