stevenzwu commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4517841806
##########
format/spec.md:
##########
@@ -157,6 +157,7 @@ Version 4 of the Iceberg spec adds support for relative
locations in metadata, e
* **Delete manifest** -- A file that lists delete files to be associated with
data files at planning time.
* **Data file** -- A file that contains rows of a table.
* **Delete file** -- A file that encodes rows of a table that are deleted by
position or data values.
+* **Column file** -- (v4+) A file that stores the values of one or more fields
of a table or a v4 leaf manifest.
Review Comment:
"fields of a table or a v4 leaf manifest" can read as a table's fields or a
manifest's fields. A column file belongs to a data file or a leaf manifest:
```suggestion
* **Column file** -- (v4+) A file that stores the values of one or more
fields for a data file or a leaf manifest.
```
##########
format/spec.md:
##########
@@ -1748,7 +1752,7 @@ The rows in the delete file must be sorted by `file_path`
then `pos` to optimize
Equality delete files identify deleted rows in a collection of data files by
one or more column values, and may optionally contain additional columns of the
deleted row.
-Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table.
+Equality delete files must not be added to v4 tables. Equality deletes cannot
be added as an entry to a v4 manifest. Equality delete files referenced by v2
or v3 delete manifests must still be applied in a v4 table. Adding column files
to a data file raises its data sequence number, so equality delete files cannot
be applied to it (see [Writing Column Files](#writing-column-files)).
Review Comment:
"cannot be applied" reads like a new reader rule. It is a consequence of the
scan planning sequence number check:
```suggestion
Equality delete files must not be added to v4 tables. Equality deletes
cannot be added as an entry to a v4 manifest. Equality delete files referenced
by v2 or v3 delete manifests must still be applied in a v4 table. Adding column
files to a data file sets its data sequence number to that snapshot's sequence
number, so existing equality delete files no longer apply to it (see [Writing
Column Files](#writing-column-files)).
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
Review Comment:
"Align" is not defined until Reading Column Files, and the requirement here
is the same row count:
```suggestion
Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have the same row count as the file
it modifies.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
Review Comment:
"includes", "nested in", and "contains" describe the same relationship, and
"contains" is only defined for structs. A list or map in `field_ids` also
replaces everything nested in it, so the definition should cover any field:
```suggestion
A column file stores the **entire** value of each field whose ID is in its
`field_ids`, including every field nested in it. In this section, a field
_contains_ every field nested in it, at any depth.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
Review Comment:
"The table schema" does not say which schema version, and "a row lineage
field required by Writing Column Files" is only
`_last_updated_sequence_number`, so naming it is clearer.
The last sentence only says what is forbidden. Stating that a list or map is
replaced as a whole makes the allowed case explicit, including a list or map
nested in a struct, such as `s.tags`:
```suggestion
When a column file is added, each ID in its `field_ids` must identify a
field in the schema of the file it modifies, or `_last_updated_sequence_number`
(see [Writing Column Files](#writing-column-files)). The schema of a data file
is the current table schema. The schema of a leaf manifest is the tracked file
schema. A list or map can only be replaced as a whole: fields nested in a list
or map must not be in `field_ids`.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
Review Comment:
"The update" is not defined now that the terms list is gone, and the second
sentence reads as a definition rather than a rule. "A struct that contains it"
also cannot be met for a field inside a list, because the list element cannot
be in `field_ids`:
```suggestion
The `field_ids` of the added column files must have the ID of each field
whose values change, or of a field that contains it. A struct's null state is
stored with the struct, so if a struct changes between null and non-null, its
ID must be in `field_ids`.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
Review Comment:
"has the same partition values after the update" can mean the same as before
the update, or the same as each other. The second sentence implies the latter.
Putting the condition first also cuts the commas:
```suggestion
- For a data file, if an added column file's `field_ids` has the ID of a
partition source field or of a field that contains one, all live rows must
share the same partition values after the update. `partition` must be set to
those values.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
Review Comment:
"take responsibility for" is informal, and the "replacing ..." clause is
already covered by the removal rule above:
```suggestion
- For a data file, one of the added column files must have the
`_last_updated_sequence_number` ID in its `field_ids`. In that column file:
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
Review Comment:
"The file it modifies" has no clear antecedent here: "it" should be the
column file, but the subject is the projected fields. "Lists" is easy to
confuse with the list type, and "a struct that contains it" misses fields
inside a list or map in `field_ids`. For `L: list<struct<x>>` in `field_ids`, a
projection of `L.element.x` falls through to the data file:
```suggestion
Each projected field, including each field nested in a projected field, must
be read from the column file whose `field_ids` has the ID of the field or of a
field that contains it. Other projected fields are read from the base file
using [Column Projection](#column-projection). IDs in `field_ids` that are not
projected are ignored.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
+
+If a struct is null in a row, each field that the struct contains is null in
that row. This applies when the struct and the nested field are read from
different files.
Review Comment:
"This applies when" reads as "only when":
```suggestion
If a struct is null in a row, each field that the struct contains is null in
that row, even when the struct and the nested field are read from different
files.
```
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
Review Comment:
[Ryan's comment in PR
18526](https://github.com/apache/iceberg/pull/18526#discussion_r4461784544)
probably also applies here.
##########
format/spec.md:
##########
@@ -1801,6 +1805,51 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`. A listed struct
includes all fields nested in it. In this section, a struct _contains_ a field
if the field is nested in the struct at any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field
in the schema of the file it modifies, or a row lineage field required by
[Writing Column Files](#writing-column-files). The schema of a data file is the
table schema. The schema of a leaf manifest is the tracked file schema. Fields
nested in a list or map must not be listed in `field_ids`.
+
+#### Writing Column Files
+
+A column file stores the value of each of its listed fields for every row of
the file it modifies. Values that did not change are still included, and values
for deleted rows should be null.
+
+The added column files must list each field that the update changes, or a
struct that contains that field. If an update changes whether a struct is null,
the update changes that struct.
+
+When column files are added to a data file or a leaf manifest, the writer must
produce a MODIFIED entry and a REPLACED entry as described in [Updating
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be
the previous entry's (inherited) snapshot ID. For a leaf manifest,
`tracking.sequence_number` must not change. The fields of the MODIFIED entry
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+ - For a data file, an added column file may list a field that is, or
contains, a source field of the data file's partition spec only if every row
that is not deleted has the same partition values after the update. `partition`
must be set to those values.
+- `sort_order_id`:
+ - If an added column file lists a field that is, or contains, a field in
the sort order, `sort_order_id` must be null.
+- `content_stats`:
+ - Stats for a field listed in an added column file, and for its nested
fields, must be computed from that column file, or omitted.
+ - For a leaf manifest, stats that summarize a listed field must be
recomputed or omitted.
+- `tight_stats`:
+ - `tight_stats` may be true only if `content_stats` are tight for the rows
that are live after the update (see [Stats Tightness](#stats-tightness)).
+- `deletion_vector`:
+ - For a data file, every delete that applied to it before the column files
were added must be written into the MODIFIED entry's `deletion_vector`.
+- `column_files`:
+ - Each field listed in an added column file must be removed from the
`field_ids` of the previous entry's column files. Each field that a listed
struct contains must also be removed. A column file with no remaining
`field_ids` must be removed.
+ - For a data file, one of the added column files must take responsibility
for `_last_updated_sequence_number` by listing it in its `field_ids`, replacing
the data file or previous column file that stored it. In that column file:
+ - If the added column files change the row,
`_last_updated_sequence_number` must be set to `null` (so that the sequence
number of the snapshot that added the column files replaces the current value).
+ - If the added column files do not change the row and the row is not
deleted, the existing `_last_updated_sequence_number` value, with null resolved
as in [Row lineage assignment](#row-lineage-assignment), must be copied to the
new column file.
+
+#### Reading Column Files
+
+A projected field must be read from the column file that lists the field or a
struct that contains it. This applies to each field nested in a projected
struct. All other projected fields are read from the file it modifies, using
[Column Projection](#column-projection) for a data file. IDs in `field_ids`
that are not projected are ignored.
+
+If a struct is null in a row, each field that the struct contains is null in
that row. This applies when the struct and the nested field are read from
different files.
+
+If a column file does not store a field that must be read from it, the reader
must use the field's `initial-default`, or null if no default is set.
Review Comment:
It is not clear how this happens, since a column file stores every field in
its `field_ids`. This seems like the case: a nested field is added to a field
that is already in a column file's `field_ids`, such as adding `s.c` after a
column file with `s` was written. Naming it makes the rule easier to follow:
```suggestion
If a field is added to the table schema after a column file is written, and
a field in that file's `field_ids` contains it, the column file does not store
it. The reader must use the field's `initial-default`, or null if no default is
set.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
