emkornfield commented on code in PR #18526: URL: https://github.com/apache/iceberg/pull/18526#discussion_r4288770323
########## format/spec.md: ########## @@ -1071,6 +1077,25 @@ Inheriting sequence numbers through the metadata tree allows writing a new manif Inheritance does not apply to v1, which does not have inherited fields like sequence number. When reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0. +##### Commit Timestamp Inheritance + +In v4 and later, the `snapshot_timestamp` tracking field records the commit time (the snapshot's `timestamp-ms` converted to microseconds) of the snapshot that added or last modified a file's data, and is used to assign `_last_updated_time` for [row lineage](#row-lineage). + +The snapshot's `timestamp-ms` is not known until the snapshot is successfully committed and may change when a commit is retried, so writers set `snapshot_timestamp` to `null` when the value must come from the committing snapshot. Writers must set `snapshot_timestamp` for data file and leaf manifest entries as follows: + +* An ADDED entry must have a `null` `snapshot_timestamp`. +* An entry MODIFIED by column files must have a `null` `snapshot_timestamp`. +* Any other entry written to a new manifest must have its inherited or previously persisted `snapshot_timestamp`. + +When reading, a `null` commit timestamp of an ADDED or MODIFIED entry is inherited: + +* For an entry in the root manifest, from the snapshot's `timestamp-ms` converted to microseconds. +* For an entry in a leaf manifest, from the inherited or persisted `snapshot_timestamp` of the root manifest entry that tracks the leaf manifest. + +For an entry with any other status, a `null` commit timestamp is not inherited and indicates that the commit timestamp is unknown. + +Files and manifests added before a table was upgraded to v4 have an unknown commit timestamp. Manifests written before v4 do not have a `snapshot_timestamp` field and their entries must be read with a `null` commit timestamp. When writing an entry that was added before the upgrade to a new manifest, writers must write a `null` `snapshot_timestamp`. If such an entry is MODIFIED because its deletion vector changed, its `null` commit timestamp is inherited when reading. Review Comment: do we want to allow for backfilling of if snapshot information is still around in metadata.json. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
