abstractdog opened a new pull request, #2887:
URL: https://github.com/apache/datafusion-ballista/pull/2887

   # Which issue does this PR close?
   
   Part of #1986. Addresses the hang reported in #2539, which this turns into a 
job failure; see below for what it leaves open.
   
   # Rationale for this change
   
   With AQE, a job hangs forever once it loses the shuffle output of a stage 
that has already finished. I hit this on `main` (6ee64d64) running TPC-DS 
SF10000 on 12 executors, with push-staged scheduling: several executors were 
evicted mid-query, and q40 then sat with no running tasks until a client-side 
timeout cancelled it 27 minutes later. The scheduler log matches #2539:
   
   ```
   WARN  state::aqe: Need to resubmit the current running Stage 2 and its map 
Stage 0 due to FetchPartitionError from task TID 038FAGQ4P8000/2.0/6
   WARN  state::aqe: Reset 1 tasks for successful job/stage 038FAGQ4P8000/1 on 
lost Executor b152409e-...
   WARN  state::task_manager: Failed to update task statuses for job 
038FAGQ4P8000, skipping its updates: DataFusion error: Execution error: Can't 
find active stage to update stage outputs
   ```
   
   So the hang does not depend on pull-staged mode.
   
   The cause: when a stage finishes, the `AdaptivePlanner` drops it from 
`runnable_stage_cache` and `runnable_stage_output`. Losing that stage's output, 
through a lost executor (`reset_stages_on_lost_executor`) or a reduce task's 
`FetchPartitionError`, makes the graph call `rerun_successful_stage`. That 
turns the stage back into a running one in the graph only. The first rerun task 
to report then fails in `update_exchange_locations` with "Can't find active 
stage to update stage outputs". The task manager drops the whole status update 
on that error, so the stage never completes. Even without that error, 
`update_stage_progress` only adds stages the graph does not have yet, so the 
rolled-back reader stage would never be rebuilt.
   
   Rerunning a finished stage under AQE is the missing recovery that #1986 
tracks. Until it exists, failing the job with a clear message is better than 
hanging it: the client gets an error it can act on, and the job releases its 
resources.
   
   # What changes are included in this PR?
   
   - `reset_stages_on_lost_executor` returns an error when the lost executor 
held output that a finished stage's readers still need. 
`TaskManager::executor_lost` already fails the job when this returns an error.
   - `processing_stages_update` turns each finished stage that a fetch failure 
would resubmit into a failed stage, so the job fails through the usual path.
   - Both report the same message: the stage's output was lost, AQE cannot 
rerun a finished stage yet, so retry the query or set 
`ballista.planner.adaptive.enabled` to `false` to use the static planner, which 
recovers lost output.
   - `AdaptiveExecutionGraph::rerun_successful_stage` stays, as the 
`ExecutionGraph` trait requires it, but the adaptive graph no longer calls it.
   - Tests in `aqe/test/job_failure.rs` for both paths. 
`losing_a_shared_stage_output_rolls_back_the_stage_once` in 
`aqe/test/reuse_exchange.rs` recorded the old behaviour; it is renamed to 
`losing_a_shared_stage_output_fails_the_rollback` and now expects the error. 
All three fail without the change.
   
   Not addressed here: actually rerunning the stage (#1986), and the stale 
tasks #2539 reports in pull-staged mode, which a rolled-back stage cannot 
cancel.
   
   # Are there any user-facing changes?
   
   Yes. An AQE job that loses the output of a finished stage now fails with the 
error above instead of hanging. The AQE limitations in the tuning guide say so. 
Losing only the output of running stages is handled as before.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to