dramaticlly commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4622500301


##########
format/spec.md:
##########
@@ -157,6 +157,7 @@ Version 4 of the Iceberg spec adds support for relative 
locations in metadata, e
 * **Delete manifest** -- A file that lists delete files to be associated with 
data files at planning time.
 * **Data file** -- A file that contains rows of a table.
 * **Delete file** -- A file that encodes rows of a table that are deleted by 
position or data values.
+* **Column file** -- (v4+) A file that stores the values of one or more fields 
for a data file or a leaf manifest.

Review Comment:
   shall we include column file in `### Manifests` section for leaf manifest at 
LINE 810?



##########
format/spec.md:
##########
@@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. 
Readers must ignore fields in a column file whose IDs are not in its 
`field_ids`. In this section, a field _contains_ every field nested in it, at 
any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
in the schema of its base file, or `_last_updated_sequence_number` (see 
[Writing Column Files](#writing-column-files)). The schema of a data file is 
the current table schema. The schema of a leaf manifest is the tracked file 
schema.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that contains the field is non-null after the update.

Review Comment:
   is this null/non-null comes from storage value or the reader derived value?  
How about a column update for a struct nested in a null ancestor, do we keep 
base file values ?



##########
format/spec.md:
##########
@@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. 
Readers must ignore fields in a column file whose IDs are not in its 
`field_ids`. In this section, a field _contains_ every field nested in it, at 
any depth.

Review Comment:
   > A column file may also contain fields that are not in its `field_ids` 
   
   
   seem to imply that a column file may also contain fields that are not in its 
`field_ids`, not sure if this relate to parquet for empty struct group problem 
but worth state it explicitly. 



##########
format/spec.md:
##########
@@ -1801,6 +1804,79 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. 
Readers must ignore fields in a column file whose IDs are not in its 
`field_ids`. In this section, a field _contains_ every field nested in it, at 
any depth.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
in the schema of its base file, or `_last_updated_sequence_number` (see 
[Writing Column Files](#writing-column-files)). The schema of a data file is 
the current table schema. The schema of a leaf manifest is the tracked file 
schema.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that contains the field is non-null after the update.
+
+- A field nested in a list or map must not be in `field_ids`. When such a 
field changes, the list or map must be in `field_ids`.
+- Each field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When column files are added to a data file or a leaf manifest, the writer must 
produce a MODIFIED entry and a REPLACED entry as described in [Updating 
Tracking Metadata](#updating-tracking-metadata). The MODIFIED entry and the 
REPLACED entry may be written to different manifests.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:
+    - For a data file, an added column file's `field_ids` may have the ID of a 
partition source field, or of a field that contains one, only if all live rows 
share the same partition values after the update. `partition` must be set to 
those values.

Review Comment:
   would it help to clarify further on the partition source field, like if it's 
relate to table's latest partition spec or the spec_id belong to the file at 
the time of writing. It shall help cover the partition evolution case better, 
especially for field which is later removed from table's partition spec. Same 
for the sort_order_id below 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to