huaxingao commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4500327095
########## format/spec.md: ########## @@ -1801,6 +1804,43 @@ If a delete column in an equality delete file is later dropped from the table, i Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. +### Column Files + +Starting in v4, a data file or v4 leaf manifest may store fields in column files listed in [`column_files`](#column-file-fields). A column file can use any supported Iceberg file format and must have rows that align with the file it modifies. + +A column file stores the fields listed in its `field_ids`, each as a whole, including all of its nested fields. Each ID must identify a field in the schema of the file it modifies, which is the table schema for a data file and the tracked file schema for a leaf manifest, or a row lineage field required by [Writing Column Files](#writing-column-files). Fields nested within a list or map must not be listed in `field_ids`. Review Comment: > Each ID must identify a field in the schema of the file it modifies Question: what happens when a field is dropped after a column file lists it? For example, a column file lists `[7]` (`customer.email`) and later the table drops `email`. Then ID 7 no longer identifies a field in the table schema, so just dropping a column would break this "must". Writers won't rewrite manifests to clean it up. Should this say the ID must identify a field in the schema when the column file is added, and readers ignore IDs that are not in the scan schema? That would match how data files keep dropped columns today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
