amogh-jahagirdar commented on code in PR #16025:
URL: https://github.com/apache/iceberg/pull/16025#discussion_r4273154240
##########
format/spec.md:
##########
@@ -1130,6 +1276,14 @@ When expiring snapshots, retention policies in table and
snapshot references are
2. The snapshot is not one of the first `min-snapshots-to-keep` in the
branch (including the branch's referenced snapshot)
5. Expire any snapshot not in the set of snapshots to retain.
+#### Deleting Files
+
+A file may be deleted from the file system when the snapshot in which it was
deleted is garbage collected, assuming that older snapshots have also been
garbage collected [1].
+
+Notes:
+
+1. Technically, data files can be deleted when the last snapshot that contains
the file as "live" data is garbage collected. But this is harder to detect and
requires finding the diff of multiple snapshots. It is easier to track what
files are deleted in a snapshot and delete them when that snapshot expires. It
is not recommended to add a deleted file back to a table. Adding a deleted file
can lead to edge cases where incremental deletes can break table snapshots.
Review Comment:
Though I see what you mean, the wording suggests that it's recommended to do
an incremental cleanup all the time but for branching that's wrong. It's
existing wording though but maybe there's a simple way to clean this up by just
saying,
> Technically, data files can be deleted when the last snapshot that
contains the file as "live" data is garbage collected.
This sentence is still true.
> It is not recommended to add a deleted file back to a table. Adding a
deleted file can lead to edge cases where incremental deletes can break table
snapshots.
This final sentence is still true as well.
>But this is harder to detect and requires finding the diff of multiple
snapshots. It is easier to track what files are deleted in a snapshot and
delete them when that snapshot expires.
This part is still techncially true but reads as a strong suggestion where
that suggestion is unsafe for branches/tags, just needs to be worded in a way
where it makes it clear that it's conditional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
