abernardi597 commented on code in PR #16778:
URL: https://github.com/apache/lucene/pull/16778#discussion_r4349999135
##########
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java:
##########
@@ -132,16 +133,46 @@ public OnHeapHnswGraph build(int maxOrd) throws
IOException {
(rebalanceEndNs - rebalanceStartNs) / 1_000_000.0));
}
}
- List<Callable<Void>> futures = new ArrayList<>();
- for (int i = 0; i < workers.length; i++) {
+
+ // Reserve first worker for calling thread: only create Futures for
remaining workers
+ List<Callable<Void>> tasks = new ArrayList<>(workers.length - 1);
+ for (int i = 1; i < workers.length; i++) {
int finalI = i;
- futures.add(
+ tasks.add(
() -> {
workers[finalI].run(maxOrd);
return null;
});
}
- taskExecutor.invokeAll(futures);
+
+ final List<Future<Void>> started = new ArrayList<>(workers.length);
+ try {
+ int start = workers[0].getStartPos(maxOrd);
+ int end;
+ while (start != -1) {
+ // Use this thread for the first worker
+ end = start + workers[0].batchSize;
+ if (end >= maxOrd) {
+ end = maxOrd; // No need to recruit more workers if this is the last
batch
+ } else {
+ // Recruit other threads for workers that haven't started yet
+ while (started.size() < tasks.size()) {
+ var maybeFuture =
taskExecutor.tryInvoke(tasks.get(started.size()));
Review Comment:
The `cancel` call only prevents the jobs from executing if they haven't
already started: it won't interrupt in-progress ones.
But this is a good idea! We can likely short-circuit out by consuming the
remaining work in the failing worker thread and propagate the failure(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
