gtrettenero opened a new pull request, #16336: URL: https://github.com/apache/iceberg/pull/16336
## Summary - Defer manifest rewriting in `BaseRewriteManifests.addManifest` by collecting manifests that require copy/rewrite into a pending list. - Perform those manifest copies in parallel during `apply()` using the existing `workerPool()` via `Tasks.range(...)`. - Preserve deterministic manifest ordering by writing parallel results into an index-addressed array before appending to `rewrittenAddedManifests`. - Keep existing commit semantics and validation behavior unchanged; this primarily removes driver-side serial copy overhead during manifest rewrite commits (especially when `compatibility.snapshot-id-inheritance.enabled=false`). ## Why `RewriteManifests` could spend significant time copying added manifests serially on the driver. This change parallelizes that copy step while preserving output correctness and ordering. ## Test Plan The logic should be the same, it should just run in parallel so it should be faster. - [x] `./gradlew :iceberg-core:test --tests org.apache.iceberg.TestRewriteManifests` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
