huaxingao commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4498686048
##########
format/spec.md:
##########
@@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`, each as a whole,
including all of its nested fields. Each ID must identify a field in the schema
of the file it modifies, which is the table schema for a data file and the
tracked file schema for a leaf manifest, or a row lineage field required by
[Writing Column Files](#writing-column-files). Fields nested within a list or
map must not be listed in `field_ids`.
Review Comment:
+1, I think these rules are right and we need them. The current text has two
problems with structs:
1. Reading: L1811 says a listed struct includes its nested fields, but L1840
only reads a field from a column file if its own ID is listed. So with
`field_ids: [2]`, `customer.name` (3) would be read from the data file.
2. Writing: if column file A lists `[3]` and a later update adds column file
B with `[2]`, L1833 only removes ID 2, so A keeps `[3]`. The reader then reads
the old `name` from A. Your change to also remove the fields a listed struct
contains fixes this.
One small suggestion: could we put "no other column file of that entry can
list a field that the struct contains" next to the rule in the `column_files`
field section ("A field ID may appear in `field_ids` of at most one column
file")? Then the overlap rules are in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
