amogh-jahagirdar commented on code in PR #16025: URL: https://github.com/apache/iceberg/pull/16025#discussion_r4038362523
########## format/spec.md: ########## @@ -656,15 +662,33 @@ A data or delete file is associated with a sort order by the sort order's id wit ### Manifests -A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. +A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a snapshot root for each table snapshot. In v4, the snapshot root is a root manifest that may track data files in addition to leaf manifest files. A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. -A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. +Each manifest type contains the following content: -A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest. +| Manifest type | Contents | +|----------------|----------| +| v1-v3 data manifest | Data files | +| v2-v3 delete manifest | Delete files | +| v4 root manifest (snapshot root) | Data files, data manifests, delete manifests | +| v4 data manifest | Data files and their colocated deletion vectors | -A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata: +In v2-v3, data and delete files are kept in separate manifests because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. + +**Partition Spec Binding:** + +- v1-v3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. +- v4: A manifest may store files written with different partition specs. Review Comment: Since we want to drop the partition tuple, I left this out -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
