gaborkaszab commented on code in PR #16425: URL: https://github.com/apache/iceberg/pull/16425#discussion_r4641476970
########## format/spec.md: ########## @@ -1434,11 +1437,12 @@ When expiring snapshots, retention policies in table and snapshot references are #### Deleting Files -A file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that the file is not live in any remaining snapshot [1]. +A file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that the file is not live in any remaining snapshot [1], [2]. Notes: 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. When all snapshots are ancestors of a single branch, it is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots. +2. A column file removed from `column_files` is not recorded with a DELETED entry. It may be deleted from the file system when the last snapshot that still references it, including on a REPLACED entry, is garbage collected. The same rule applies to a deletion vector that remains only on a REPLACED entry. Review Comment: Shouldn't we clarify more, that when the snapshot that has it in a REPLACED entry, but the respective MODIFIED entry doesn't have it, then we can physically erase the column file? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
