grantatspothero commented on code in PR #18156:
URL: https://github.com/apache/iceberg/pull/18156#discussion_r4047988053
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -800,7 +800,8 @@ private DeleteFileIndex buildDeleteFileIndex(
DeleteFileIndex.builderFor(ops().io(), deleteManifests)
.afterSequenceNumber(startingSequenceNumber)
.caseSensitive(caseSensitive)
- .specsById(ops().current().specsById());
+ .specsById(ops().current().specsById())
+ .planWith(workerPool());
Review Comment:
One thing that came up is reentrancy of parallelizing tasks using the worker
pool. See discussion here:
https://github.com/apache/iceberg/pull/18076#discussion_r4007880119
In this example, it means we cannot parallelize both across snapshots and
within a snapshot due to reentrancy and deadlock. So you have to choose which
level to parallelize at, and I think I agree with you parallelizing across
snapshots is better.
Is this reentrancy issue something you have seen elsewhere in Iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
