RussellSpitzer commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4418369076
########## format/spec.md: ########## @@ -1801,6 +1804,40 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file may store fields in column files listed in [`column_files`](#column-file-fields). A column file is an Iceberg data file and must have the same row count as its data file. + +A column file stores the entire value of each field in its `field_ids`. Each ID must identify a field in the table schema, or a row lineage field required by [Writing Column Files](#writing-column-files). A schema field must be top-level or nested in structs, must not be nested within a list or map, and must not be, or contain, a source field of the data file's partition spec. + +#### Writing Column Files + +In this subsection, the **previous entry** is the live entry of the data file before column files are added, an **updated field** is a table schema field whose ID is in the `field_ids` of an added column file, and an **updated row** is a row whose updated-field values were set by the update. Other rows keep their current values. + +A new column file stores the value of each of its listed fields for every row of the data file, including rows the update did not change. Values for deleted rows should be null, so every field in `field_ids` must be optional in the column file's schema. Review Comment: ```suggestion A column file stores the value of each of its listed fields for every row of the data file. Values which did not changed should still be included and values for deleted rows should be null,. ``` I think you can ignore the "file's schema" note since that's an implentation note I think. We don't actually care what the file's schema is other than it matches the type of the original schema but even that seems like a bit of a detail? Maybe adding a line later, "fields should always be read as the type of the field in the table schema" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
