andygrove opened a new issue, #2539:
URL: https://github.com/apache/datafusion-ballista/issues/2539
## Describe the bug
With AQE enabled, a job hangs forever after losing **one** executor out of
32. Every other executor stays healthy. The scheduler starts recovering the
lost shuffle output, but:
1. It cannot cancel the now-stale tasks of the rolled-back stage in
pull-staged mode, so they keep running and reporting.
2. A second `FetchPartitionError` hits the resubmitted map stage. Updating
its outputs then fails with `Can't find active stage to update stage outputs`.
After that the job never progresses and never fails. The client waits
indefinitely.
This is related to #2029 but is not the same bug. #2029 covers losing
*every* executor. Here a single executor is lost while 31 healthy executors
remain, so the job should recover.
## Environment
- Ballista Scheduler/Executor v55.0.0 (DataFusion v55.1.0), as reported in
the startup logs
- Kubernetes, 1 scheduler + 32 executors, `--vcores 8` (passed as the
deprecated `--concurrent-tasks 8`), `--task-scheduling-policy pull-staged`
- AQE enabled (plans show `AdaptiveDatafusionExec`, and the log lines come
from `state::aqe`)
- Sort-based shuffle with executor work dir on local disk
- TPC-H SF1000 Parquet on S3, driven by the upstream `tpch benchmark
ballista` binary
- The hang was on Q5. Q1–Q4 completed normally in the same cluster.
## Timeline (scheduler log, trimmed)
The executor pod was sent SIGTERM by Kubernetes (most likely node
scale-down). It was not OOM-killed.
```
17:31:45 Job [03887Z2ZZJ000] planning took 375.496014ms
17:31:45 Job submitted: [03887Z2ZZJ000]
17:32:04 INFO scheduler_server::grpc: Received executor stopped request
from Executor e3dc5e6e-... with reason 'executor received terminate signal.'
17:32:34 INFO state::executor_manager: Removing executor e3dc5e6e-...:
Some("executor received terminate signal.")
17:32:34 WARN state::aqe: Reset 1 tasks for running job/stage
03887Z2ZZJ000/3 on lost Executor e3dc5e6e-...
17:33:07 WARN state::aqe: Need to resubmit the current running Stage 4 and
its map Stage 2 due to FetchPartitionError from task TID 03887Z2ZZJ000/4.0/2
17:33:07 ERROR state::executor_manager: Failed to get client for executor ID
13edcc1c-... to cancel tasks
17:33:07 ERROR state::executor_manager: Failed to get client for executor ID
abba0495-... to cancel tasks
... the same error for every one of the ~31 live executors ...
17:33:07 WARN state::aqe: Stage 03887Z2ZZJ000/4 is not in running when
updating the status of tasks [11]
... ~30 more "Stage 03887Z2ZZJ000/4 is not in running" warnings as the
uncancelled stage-4 tasks finish ...
17:35:40 WARN state::aqe: Need to resubmit the current running Stage 2 and
its map Stage 0 due to FetchPartitionError from task TID 03887Z2ZZJ000/2.1/34
17:35:40 ERROR query_stage_scheduler: Failed to update 1 task statuses for
Executor b5884438-...: DataFusionError(Execution("Can't find active stage to
update stage outputs"))
17:35:47 WARN state::aqe: Need to resubmit the current running Stage 2 and
its map Stage 0 due to FetchPartitionError from task TID 03887Z2ZZJ000/2.1/33
17:35:47 ERROR query_stage_scheduler: Failed to update 1 task statuses for
Executor 13ed3042-...: DataFusionError(Execution("Can't find active stage to
update stage outputs"))
17:40:23 WARN state::aqe: Stage 03887Z2ZZJ000/4 is not in running when
updating the status of tasks [14]
... nothing further for this job; it never finishes or fails ...
```
On a surviving executor, the stale stage-4 task ran to completion and failed
on the fetch from the dead executor:
```
17:33:10 INFO ballista_executor: Task 30 failed: DataFusion error: External
error: Shuffle fetch partition error from Executor e3dc5e6e-..., map_stage 2,
map_partition 242, error desc: Error connecting to Ballista scheduler or
executor at http://<dead-executor-ip>:50051: tonic::transport::Error(Transport,
ConnectError(ConnectError("tcp connect error", ..., Os { code: 113, kind:
HostUnreachable, message: "No route to host" })))
```
## Observations from the code
- **Task cancellation in pull-staged mode.**
`ExecutorManager::cancel_running_tasks` (`state/executor_manager.rs`) calls
`get_client` for each executor. In pull-staged mode the scheduler has no
clients for executors, so every cancellation fails with `Failed to get client
for executor ID ... to cancel tasks`. The tasks of the rolled-back stage keep
running and report status for a stage that is no longer running.
- **Second fetch failure during the resubmit.** At 17:35:40 a task from the
resubmitted map stage (`2.1`) hits another `FetchPartitionError`, which
triggers a rollback of Stage 2 / map Stage 0. The task status update then fails
in `AdaptivePlanner::update_exchange_locations` (`state/aqe/planner.rs`)
because `runnable_stage_output` has no entry for the stage. The error
propagates out of `update_task_status`, and the job appears to be left with no
stage runnable, no stage marked failed, and nothing that would ever fail the
job.
## Expected behavior
Losing one executor while others remain should either:
- recover: re-run the lost map outputs on the remaining executors and finish
the query, or
- if recovery is impossible, fail the job with a clear error once the
stage-failure limit is reached.
It should never hang. Also, in pull-staged mode, tasks of a rolled-back
stage should be discarded cleanly. The scheduler can't push cancellations, but
it could tell executors to cancel when they next poll.
## To reproduce
This should be reproducible with the chaos harness from #2026/#2030. Run a
multi-stage join query (e.g. TPC-H Q5) with AQE enabled and the pull-staged
policy on enough executors that many stages run at once. Then SIGTERM a single
executor once the second or third stage is running. A run where the resubmitted
map stage itself hits a fetch failure, for example because more of its inputs
were on the lost executor, should hit the `Can't find active stage to update
stage outputs` path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
