amogh-jahagirdar commented on code in PR #18156:
URL: https://github.com/apache/iceberg/pull/18156#discussion_r4047662274
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -800,7 +800,8 @@ private DeleteFileIndex buildDeleteFileIndex(
DeleteFileIndex.builderFor(ops().io(), deleteManifests)
.afterSequenceNumber(startingSequenceNumber)
.caseSensitive(caseSensitive)
- .specsById(ops().current().specsById());
+ .specsById(ops().current().specsById())
+ .planWith(workerPool());
Review Comment:
This is parallelizing delete manifest reads within a single DeleteFileIndex
build (i.e. useful when there are many delete manifests added in a given
snapshot).
But if we want to optimize the case where many snapshots were added, we
would want to parallelize the creation of all these indices
https://github.com/apache/iceberg/blob/ee7ca67874313135a9badded546ecfe5ba770789/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L664
.
I think optimizing for the latter is more useful because reducing validation
latency is more useful for things like compaction jobs trying to keep up with
streaming workloads where the number of delete manifests that should be
produced per commit should be small to begin with (and even for big batch jobs,
we'd be merging delete manifests etc) so parallelism at that level in
validation probably isn't that useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
