raunaqmorarka opened a new pull request, #18074: URL: https://github.com/apache/iceberg/pull/18074
After a successful commit, `SnapshotProducer.commit()` refreshes the table and then reads the new snapshot's manifest list back from storage to find the manifests to keep during `cleanUncommitted`. `BaseTransaction` does the same for every snapshot the transaction created. On object stores each read is a GET, plus a HEAD for the length with `S3FileIO`, for a file this process wrote moments earlier. `SnapshotProducer` now keeps the `ManifestFile` list it wrote to each manifest list, keyed by manifest list location, and uses it for cleanup. The transaction path asks its updates for the same lists. Both fall back to reading the manifest list when the committed location was not written by this process, such as a snapshot fast-forwarded by cherry-pick or a concurrent snapshot committed between transaction retry attempts. Keying by location keeps retries correct when an earlier attempt's manifest list is the one that got committed. This keeps the approach from #15511: the committed snapshot is still resolved from the refreshed table state, and the in-memory manifests are only used when that state's manifest list location is one this producer wrote. Unlike #10523, cleanup still runs, and any manifest list not written by this process is read as before. The in-memory objects are the same `GenericManifestFile` instances the parser returns, and every `cleanUncommitted` in core compares by path only, so cleanup receives the same input as before. `ops.refresh()` is unchanged. Manifest list opens per operation on a freshly loaded table, measured with a counting `FileIO` through `HadoopCatalog`: | Operation | Before | After | |---|---|---| | MergeAppend commit | 2 | 1 | | Transaction with two appends | 3 | 2 | Tests assert that the committed manifest list is never opened during commit, and that retried and concurrent commits keep the right manifests. AI assistance was used to draft the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
