rdblue commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4648851864


##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.

Review Comment:
   I think there should be a weaker requirement here that is the same as other 
file reads. We don't need to state what is valid, only how to interpret the 
data: When reading, a field must be read from the column file where its field 
ID listed in `field_ids`. If no column file contains the field, it is assumed 
to be in the base file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to