adarshsanjeev commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1039001996
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>>
inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
Review Comment:
Changed
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>>
inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Yes, the future that is added as a callback should still get executed if the
completable future is already finished executing, though the javadoc doesn't
guarantee which thread would be executing it if an executor is also not passed
as an argument.
It is also not a hard necessity that we cancel every pending future but it
depends on number of workers which could be up to a 1000, which would waste a
lot of time, but correctness would still be present.
Also, with regard to retrying, part of this will be updated with the worker
retry PR, to only fetch the statistics which are needed even if a worker
restarts, so after that we would be retrying the stats collection.
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +120,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>>
inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Set<Future<?>> futuresToCancel = ConcurrentHashMap.newKeySet();
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
