RussellSpitzer commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4498275877
##########
format/spec.md:
##########
@@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+Starting in v4, a data file or v4 leaf manifest may store fields in column
files listed in [`column_files`](#column-file-fields). A column file can use
any supported Iceberg file format and must have rows that align with the file
it modifies.
+
+A column file stores the fields listed in its `field_ids`, each as a whole,
including all of its nested fields. Each ID must identify a field in the schema
of the file it modifies, which is the table schema for a data file and the
tracked file schema for a leaf manifest, or a row lineage field required by
[Writing Column Files](#writing-column-files). Fields nested within a list or
map must not be listed in `field_ids`.
Review Comment:
I don't think this is fully specified yet.
So given
```
1: id: long
2: customer: struct
3: name: string
4: address: struct
5: street: string
6: city: string
7: email: string
8: tags: list<string> -- element cannot appear in field_ids
```
And Column Files
column_files = [{ field_ids: [2] }]
Where does a reader read customer.name? Or is this an illegal "column file"?
I feel like we need to be a bit more through here.
I believe these are the rules we need if we aren't going to have resolution
rules
1. A column file must not list a field nested in a list or map.
2. A column file that lists a struct stores all fields nested in that struct.
3. If a column file of an entry lists a struct, no column file of that entry
can list a field nested in that struct.
4. An added column file must list each field that the update changes, or a
struct that contains it. If an update changes whether a struct is null, the
update changes that struct.
5. When a writer adds a column file, the writer must remove each listed
field from the column files of the previous entry. The writer must also remove
each field nested in a listed field. The writer must remove a column file that
has no remaining fields.
6. The reader must read a field from the column file that lists the field or
a struct that contains it. If no column file lists either one, the reader must
read the field from the data file or leaf manifest.
7. If a struct is null in a row, each field nested in that struct is null in
that row.
The TLDR;
You can modify any field that is not inside a list or map. Listing a struct
takes over the whole struct, including all of its children. If an update
changes whether a struct is null, the update must list that struct. If another
column file already lists a struct, you cannot list only one of its children.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
