andygrove opened a new issue, #2226: URL: https://github.com/apache/datafusion-ballista/issues/2226
**Describe the bug** `killing_every_executor_terminates_the_job::case_2_aqe_on` (`chaos-testing/tests/ha.rs`) intermittently fails on CI: the query never terminates, and the test's 120s guard fires. This is the same hang that #2029 describes, surviving the fix in #2212 — but only on the AQE-on case, and only sometimes. The AQE-off case passed in the same run, in about 6 seconds. Observed on the `macos test` job of https://github.com/apache/datafusion-ballista/actions/runs/30954222353/job/92143349419 (a PR run whose changes are unrelated to scheduling): ``` ---- killing_every_executor_terminates_the_job::case_2_aqe_on stdout ---- thread 'killing_every_executor_terminates_the_job::case_2_aqe_on' panicked at chaos-testing/tests/ha.rs:410:10: job must terminate, not hang, after every executor is killed: Elapsed(()) test result: FAILED. 9 passed; 1 failed; 6 ignored; 0 measured; 0 filtered out; finished in 150.45s ``` `Elapsed(())` is the `tokio::time::timeout(Duration::from_secs(120), query)` in the test, so the client future was still pending 120s after both executors were `SIGKILL`ed, with the harness grace period turned down to about a second. This appears to be the first occurrence — I checked the last 25 `rust.yml` runs and every other failure was a different job or a different test. So the failure is intermittent rather than a straightforward breakage. **To Reproduce** ```sh cargo test -p ballista-chaos --test ha killing_every_executor_terminates_the_job ``` Expect to need many iterations to hit it. It reproduced on a macOS runner, where the executors are slower to start and reap, so the timing is likely load-dependent. **Expected behavior** Once the last executor is reaped and the grace period elapses, the scheduler fails the job with an error naming the executor loss, under AQE on and AQE off alike. The test should pass deterministically. **Additional context** One hypothesis worth checking first, from reading the `ExecutorLost` handler in `query_stage_scheduler.rs`: the no-executors check is single-shot per burst. `no_executor_check_pending` collapses the storm of `ExecutorLost` events into one timer, and after the grace period the timer walks `get_running_job_cache()` and skips (`continue`) any job whose live status is not `Running` at that instant, then clears the flag. If AQE is mid-replan when the timer fires, so the job is momentarily not reported as `Running`, the job is skipped — and because both executors are already gone, no further `ExecutorLost` event will ever arrive to schedule a second check. The job would then hang forever. That is consistent with the failure being AQE-on-only and timing-dependent, but it is a hypothesis, not a diagnosis. Related: - #2029 is the original hang. It is still open because #2212's body has a malformed `Closes #.` reference, so GitHub never linked or auto-closed it. Worth deciding whether this issue supersedes it or whether #2029 should be closed and this tracked separately. - #2092 (AQE by default) — this is a blocker for trusting the AQE-on path under executor loss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
