rdblue commented on code in PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#discussion_r4649304155


##########
format/spec.md:
##########
@@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later 
dropped from the table, i
 
 Manifests hold the same statistics for delete files and data files. For delete 
files, the metrics describe the values that were deleted.
 
+### Column Files
+
+A data file or leaf manifest may store fields in column files listed in 
[`column_files`](#column-file-fields). The data file or leaf manifest whose 
`column_files` lists a column file is that column file's _base file_. A column 
file can use any supported Iceberg file format and must have the same row count 
as its base file.
+
+A column file stores the values of the fields whose IDs are in its 
`field_ids`. A list or map is stored as a whole, including every field nested 
in it. For a struct, the column file stores whether the struct is null. Fields 
nested in the struct are stored only if their IDs are also in `field_ids`. A 
column file may also store fields whose IDs are not in its `field_ids`. Readers 
must ignore these fields.
+
+When a column file is added, each ID in its `field_ids` must identify a field 
valid for its base file. For a data file, fields in the current table schema 
and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in 
the `tracked_file` struct are valid. Fields nested in a list or map are not 
valid.
+
+#### Writing Column Files
+
+A column file stores a value for every row of its base file, including rows 
whose values did not change. Values for deleted rows, and for fields in a null 
struct, should be null. A column file may write a field as optional even if 
that field is required in the table schema.
+
+An added column file must follow these rules for `field_ids`. A field must be 
in `field_ids` if a rule requires it for at least one row where every struct 
that the field is nested in is non-null after the update.
+
+- When a field nested in a list or map changes, the list or map must be in 
`field_ids`.
+- Each non-struct field whose value changes must be in `field_ids`.
+- Each struct that changes between null and non-null must be in `field_ids`.
+- If a struct changes from null to non-null, each field directly nested in it 
must be in `field_ids`, even if the field is null.
+
+See [Column file example](#column-file-example).
+
+When a writer adds column files to a data file, the writer must produce a 
MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking 
Metadata](#updating-tracking-metadata) for those entries. The writer may write 
the two entries to different manifests.
+
+When a writer adds column files to a leaf manifest, the writer must produce a 
MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry 
must be a full entry in the root manifest. The writer must not replace the 
previous leaf-manifest entry with a manifest deletion vector.
+
+The `tracking` fields of the MODIFIED entry are set as described in [Updating 
Tracking Metadata](#updating-tracking-metadata) and [Sequence Number 
Inheritance](#sequence-number-inheritance), and `tracking.snapshot_id` must be 
the previous entry's (inherited) snapshot ID. The fields of the MODIFIED entry 
outside `tracking` must match the previous entry, except for the following:
+
+- `partition`:

Review Comment:
   This is currently over-specified and too specific.
   
   It would be better to state that the file's metadata must be updated to 
reflect the data changes. For example, if the file is no longer sorted 
according to the order identified by `sort_order_id`, then the sort order must 
be updated.
   
   We want to avoid statements like "if X then you must Y" because those are 
rigid and must be completely correct. It's overly rigid because these changes 
don't necessarily have the required effect. If you replace a sort field but the 
rows are still in sorted order, this still requires implementations to set the 
order ID to null (**which, by the way, is not a valid sort order ID**).
   
   A better way is to state consistency requirements, or (better yet) let other 
parts of the spec state their consistency requirements. Sort order is a good 
example again: when the sort order ID is set, the file's data must be in that 
sort order. Then all you need to say here is that the metadata must be 
consistent. You can't change the data and not worry about the rest of the 
metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to