namest504 opened a new pull request, #18738: URL: https://github.com/apache/iceberg/pull/18738
Closes #18737 Since #13631, `BaseTransaction.expireSnapshots()` goes through `appendUpdate`, which only wires `deleteWith(enqueueDelete)` for `SnapshotUpdate`. `RemoveSnapshots` implements `ExpireSnapshots`, so its delete callback stays null and the expired manifest lists, manifests and data files are removed by `FileIO` when the transaction-local `commit()` runs, before anything reaches the catalog. If the process dies or `commitTransaction()` fails, the table keeps referencing snapshots whose files are gone. This wires `ExpireSnapshots` the same way and restores the behaviour from #218 and #9183: the files are queued in `deletedFiles` and deleted by `commitTransaction()`. `deletedFiles` is now a concurrent set. `ExpireSnapshots.executeDeleteWith()` runs the delete callback on the caller's executor, so the queue has to take adds from several threads. Two new tests check the state between the two commits, one with the default direct executor and one with a four thread delete executor. Both fail on main for all parameter combinations (format versions 1 to 4, incremental and reachable cleanup) and pass with the change. The existing transaction tests only look at the end state, which is why they did not catch the regression. Not covered here, and the same as before 1.10: when `commitTransaction()` fails with a `CleanableFailure`, `cleanUp()` still deletes everything in `deletedFiles`, including the expired snapshots' files. I will open a separate issue for that. Happy to open the 1.10.x and 1.11.x backports once this is in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
