rdblue commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4649118192
##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later
dropped from the table, i
Manifests hold the same statistics for delete files and data files. For delete
files, the metrics describe the values that were deleted.
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in
[`column_files`](#column-file-fields). The data file or leaf manifest whose
`column_files` lists a column file is that column file's _base file_. A column
file can use any supported Iceberg file format and must have the same row count
as its base file.
+
+A column file stores the values of the fields whose IDs are in its
`field_ids`. A list or map is stored as a whole, including every field nested
in it. For a struct, the column file stores whether the struct is null. Fields
nested in the struct are stored only if their IDs are also in `field_ids`. A
column file may also store fields whose IDs are not in its `field_ids`. Readers
must ignore these fields.
Review Comment:
This mixes a few things, like what can be in a file and how those fields
interact with `field_ids`, as well as what is allowed to be in `field_ids` and
how to interpret it.
I would start with what is in a column file and how to interpret it, using a
simple primitive field example. If possible, this should state invariants, like
"if a field ID is listed for a column file, that column file is responsible for
the field and all sub-fields". Simple rules like this help clarify other
statements, like that a map is identified by its field ID and no sub-field IDs
are listed.
Then add paragraphs on nested types. The first one should be about list,
map, file, and variant types that must be "stored whole" in one file. This
should be further restricted: list element ID, map key and value IDs, and file
sub-field IDs must not be listed in `field_ids`.
Next, add a paragraph on struct. This is a bit unclear to me right now:
> For a struct, the column file stores whether the struct is null. Fields
nested in the struct are stored only if their IDs are also in `field_ids`.
Starting with the first statement, I don't think that this states a
requirement clearly. Sounds like the intent was that the column file that lists
a struct ID is how readers should determine when a struct is null, but that
doesn't make sense. Whether a given struct is null must agree across all column
files that contain parts of the struct, so it isn't determined by just one
file. I think the struct paragraph should state that clearly: it is invalid for
column files to disagree about whether a struct is null. I think that's the
right requirement because it leaves space for changing whether struct values
are null at any level. The only requirement is that you have to ensure
consistency.
```
| File 1 | File 2 | Result |
|----------|----------|--------|
| {"a": 1} | {"b": 2} | {"a": 1, "b": 2} |
| null | {"b": 2} | Invalid |
| {"a": 1} | null | Invalid |
```
Next, I don't understand rationale for including the struct ID and the child
IDs. Is this related to the null problem? I think a simple rule is that if an
ID is listed for a file, it and any of its children not listed elsewhere are
read from that file. That results in a fairly compact representation and I
think it works in all cases. It sounds like the rule about child fields is
intended to handle the case where you have a whole struct in one file and want
to rewrite or append just one field. That would require rewriting the struct's
field IDs in the existing file, but we have to do that anyway.
Here's an example: `1: point struct<2: x double, 3: y double>`. If I want to
replace the entire point I could write `file_1.parquet` with `field_ids=[1]`
(currently this would be `field_ids=[1, 2, 3]`). To replace `x`, he result
would be `file_1.parquet / field_ids=[1]` and `file_2.parquet / field_ids=[2]`.
In the current scheme, this would be `file_1.parquet / field_ids=[1, 3]` and
`file_2.parquet / field_ids=[2]`, but I don't see how that's any more clear.
And it has me wondering why `1` is listed at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]