rdblue commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4649212780
########## format/spec.md: ########## @@ -1801,6 +1805,81 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +A data file or leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). The data file or leaf manifest whose `column_files` lists a column file is that column file's _base file_. A column file can use any supported Iceberg file format and must have the same row count as its base file. + +A column file stores the values of the fields whose IDs are in its `field_ids`. A list or map is stored as a whole, including every field nested in it. For a struct, the column file stores whether the struct is null. Fields nested in the struct are stored only if their IDs are also in `field_ids`. A column file may also store fields whose IDs are not in its `field_ids`. Readers must ignore these fields. + +When a column file is added, each ID in its `field_ids` must identify a field valid for its base file. For a data file, fields in the current table schema and `_last_updated_sequence_number` are valid. For a leaf manifest, fields in the `tracked_file` struct are valid. Fields nested in a list or map are not valid. + +#### Writing Column Files + +A column file stores a value for every row of its base file, including rows whose values did not change. Values for deleted rows, and for fields in a null struct, should be null. A column file may write a field as optional even if that field is required in the table schema. + +An added column file must follow these rules for `field_ids`. A field must be in `field_ids` if a rule requires it for at least one row where every struct that the field is nested in is non-null after the update. + +- When a field nested in a list or map changes, the list or map must be in `field_ids`. +- Each non-struct field whose value changes must be in `field_ids`. +- Each struct that changes between null and non-null must be in `field_ids`. +- If a struct changes from null to non-null, each field directly nested in it must be in `field_ids`, even if the field is null. + +See [Column file example](#column-file-example). + +When a writer adds column files to a data file, the writer must produce a MODIFIED entry and a REPLACED entry. The writer must follow [Updating Tracking Metadata](#updating-tracking-metadata) for those entries. The writer may write the two entries to different manifests. + +When a writer adds column files to a leaf manifest, the writer must produce a MODIFIED entry and a REPLACED entry in the root manifest. The REPLACED entry must be a full entry in the root manifest. The writer must not replace the previous leaf-manifest entry with a manifest deletion vector. Review Comment: What does this mean? > The writer must not replace the previous leaf-manifest entry with a manifest deletion vector. I think that "the previous leaf-manfiest entry" is referring to a leaf manifest's `TrackedFile` metadata in the root manifest. There are no manifest deletion vectors for the root manifest, so I don't know how this would happen. Another interpretation is that this is saying that a writer may either update a leaf manifest's column files OR update a leaf manifest's manifest deletion vector, but not both. I don't think this requirement makes sense. It is possible to delete files and update at the same time. For leaf manifests, we expect the most common use of column files will be replacing the DV struct for a manifest file. (This is why we included the feature for metadata.) That means we have fairly broad deletes happening, which could easily result in the rest of the rows in a file being deleted. It could also mean that some files have updated rows column replace and those entries are removed (via MDV) and moved to the root. If modifying ether the MDV OR column files is the goal, I think we have to reconsider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
