raunaqmorarka opened a new pull request, #18095: URL: https://github.com/apache/iceberg/pull/18095
After a successful commit, `BaseTransaction` reads the manifest list of every snapshot added since the `base` captured before the retry loop, to avoid deleting a committed manifest during clean-up. That set includes snapshots other writers committed while the transaction was open. Those snapshots cannot reference files this transaction wrote, because `deletedFiles` only holds manifests and manifest lists produced by the transaction's own updates. Reading them is wasted work. On S3 each manifest list read is a HEAD and a GET. With N concurrent inserts each commit reads the manifest lists of all snapshots committed since it opened, so the burst costs O(N^2) reads. Measured: 25 to 30 foreign snapshots per commit at about 35 ms each. This change takes the starting snapshot set from `base` after the retry loop. `applyUpdates` refreshes `base` on every attempt and the successful commit uses that field, so `current` minus `base` is exactly the snapshots this transaction created. A manifest deleted in one attempt and reused by a retry is still protected, because the transaction's own committed snapshot references it. Tests: - a transaction opened before concurrent commits from another table instance, with an injected retry, reads only its own manifest list after commit - a manifest merged in the first attempt but kept as is by the retry is not deleted AI assistance (Claude Code) was used to develop this patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
