fivetran-vikaspandey opened a new issue, #18132: URL: https://github.com/apache/iceberg/issues/18132
### Proposed Change ### Motivation IncrementalFileCleanup.cleanFiles (core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java) determines which manifests/files to delete by **scanning manifests across all currently-valid snapshots** (afterExpiration.snapshots(), line ~90) in addition to the expired snapshots, to detect whether a file deleted by an expiring snapshot is still referenced by any valid snapshot. This is correct in general, tables can have branches, tags, or cherry-picked/WAP commits where a valid ref's ancestry isn't a simple straight line back through the expired snapshots but for the common case of a single, purely linear history with no branches/tags/cherry-picks, this full scan does more work than necessary and **scales with total valid-snapshot count** rather than with the number of snapshots actually being expired. ### Proposed optimization Add a new opt-in flag, passed through the expireSnapshots call (e.g. via ExpireSnapshots builder API down into IncrementalFileCleanup), that when enabled, restricts the "still referenced?" **scan in cleanFiles to only the first valid snapshot after the expiry cutoff**, instead of all valid snapshots. On a linear history, if a manifest entry is marked DELETED in an expiring snapshot, the next valid snapshot after the cutoff is the earliest point where that file could still be referenced (e.g. re-added, or carried forward in a reused manifest); if it isn't referenced there, no later snapshot in a linear chain can reference it either. This is **scoped to IncrementalFileCleanup only** (not ReachableFileCleanup), and must be strictly opt-in via the flag, the caller is responsible for guaranteeing the table has no branches/tags/cherry-picked snapshots when enabling it, since IncrementalFileCleanup already has non-trivial handling for ancestor tracking, cherry-picks (pickedAncestorSnapshotIds), and reverted manifests (manifestsToRevert) that assumes full-history awareness. ### Proposal document _No response_ ### Specifications - [x] Table - [ ] View - [ ] REST - [ ] Puffin - [ ] Encryption - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
