akshat-lakhera opened a new pull request, #20803:
URL: https://github.com/apache/druid/pull/20803
Fixes #20554.
### Description
MSQWorkerFaultToleranceTest.test_cancelledWorker_isRetried_ifFaultToleranceIsEnabled
failed once in CI on 2026-10-10 (commit 792196709e, #20535, tracked on the
flaky test dashboard #20437). The failure was "Timed out waiting for event
after [10,000]ms" on the wait for the controller task to finish.
#### Root cause
After the test cancels the worker, the controller has to notice the failed
worker, relaunch it on the functional Indexer, re-read the input and publish
the segments. The test waited for all of that with waitForTaskToSucceed, which
uses the LatchableEmitter default timeout of 10 seconds, and on a busy runner
that whole sequence can take longer. I'm going by the CI log here rather than a
natural local failure: on an idle machine the same wait takes about 1.5 seconds
and I couldn't get it to fail on its own.
#### Fix
The test now waits for the controller task's task/run/time event on the
Overlord emitter with an explicit 60 second timeout, and then runs the existing
waitForTaskToSucceed and the checks after it. The longer timeout only applies
to this one wait, so the cluster-wide default stays the same, and there are no
sleeps or retries.
#### Testing
The embedded cluster doesn't start on Windows, so I ran the test in WSL
Ubuntu with JDK 25.
- Since it didn't fail on its own, I forced the slow path by adding the
functional Indexer 12 seconds after the worker failed. That hack isn't part of
this PR. Without the fix, 3 of 3 runs failed with the same "Timed out waiting
for event after [10,000]ms" error as in CI. With the fix, 3 of 3 runs passed,
with the final wait taking about 19 seconds.
- Without the forcing hack, the fixed test passed 30 of 30 runs, and the
unmodified test passed 8 of 8, so this isn't a natural before and after
comparison.
- Checkstyle and forbiddenapis pass on embedded-tests.
<hr>
##### Key changed/added classes in this PR
* MSQWorkerFaultToleranceTest
<hr>
This PR has:
- [x] been self-reviewed.
PTAL @FrankChen021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
