FrankChen021 commented on code in PR #20825:
URL: https://github.com/apache/druid/pull/20825#discussion_r4480527990
##########
multi-stage-query/src/main/java/org/apache/druid/msq/dart/controller/DartControllerRegistry.java:
##########
@@ -123,6 +132,24 @@ public void stop()
if (cleanupExec != null) {
cleanupExec.shutdown();
}
+
+ // Cancel any queries that are still running. This registry is stopped in
the NORMAL stage, after
+ // ANNOUNCEMENTS and SERVER, so by this point the Broker has been removed
from discovery and the
+ // druid.server.http.gracefulShutdownTimeout has already elapsed.
+ final List<ControllerHolder> holders = new
ArrayList<>(controllerMap.values());
+ if (!holders.isEmpty()) {
+ log.info("Canceling [%d] queries that are still running.",
holders.size());
+
+ // Cancel all controllers in sequence. It would be nicer to do this in
parallel, since it involves sending
+ // stopWorker calls to active workers. This would require some
restructuring of the code to return Futures,
+ // or an executor here used only for cancellation.
+ for (final ControllerHolder holder : holders) {
+ holder.cancel(CancellationReason.CONTROLLER_SHUTDOWN, null);
Review Comment:
[P2] Sequential cancellation bypasses the shutdown deadline
**Finding:** `DartControllerRegistry.stop()` still invokes
`holder.cancel(...)` serially. Each cancellation synchronously waits for
`DartWorkerManager.stop(true)` and its worker stop futures, which can take up
to 10 seconds, before `awaitControllerCompletion()` starts; multiple active
controllers can therefore exceed the registry's 10-second shutdown bound.
**Suggestion:** Run cancellations concurrently or propagate one shutdown
deadline through worker cancellation and completion waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
