goingforstudying-ctrl opened a new issue, #2859: URL: https://github.com/apache/datafusion-ballista/issues/2859
I was reading the executor-loss recovery path while trying to understand #2856 and found another way a job can end up hanging forever. `TaskManager::executor_lost` walks the active jobs and rolls back the stages that touched the dead executor: ```rust let reset = graph.reset_stages_on_lost_executor(executor_id)?; ``` The `?` is the problem. A rollback can fail — `rollback_resolved_shuffles` errors on plans whose nodes reject child rewrites, which the graph tests already treat as a real case (`test_resolve_stage_preserves_stage_on_plan_rewrite_error`). When that happens the error propagates out of the whole function, so: 1. the job whose rollback failed is left half-reset: `reset_stages_internal` has already marked some of its tasks failed but never rolled its running stages back, and no failure event is posted for it. Its remaining tasks stay `Running` on an executor that no longer exists, so the job never progresses and never fails; 2. every job after it in the `active_job_cache` iteration never gets its tasks reset at all, so those jobs hang too, even though their rollbacks would have succeeded. `QueryStageScheduler::on_executor_lost` only logs the returned error, so from the outside the cluster looks healthy while the jobs just stop making progress — the same "hangs instead of failing" shape as #2856. Not claiming this is what happened there; the scheduler logs needed to confirm it weren't captured. I think a rollback failure should be handled like the other "cannot survive the loss" cases already in that function (pipelined jobs, lost `DataFrame::cache()` data): fail that job with a clear reason and keep recovering the rest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
