andygrove opened a new issue, #2856:
URL: https://github.com/apache/datafusion-ballista/issues/2856
**Describe the bug**
When executors are OOM-killed by Kubernetes during a query and their
containers restart, the job can stop making progress indefinitely instead of
either recovering or failing.
Found while measuring how little executor memory TPC-H q18 at SF1000 needs
(#2855). Ballista `d9872080`, default settings (memory pool 70% of the
container, `pull-staged` scheduling, AQE on), 2 executors of 16 vcores, 32
partitions:
| Executor memory limit | Memory pool | Executor OOM kills | Outcome |
| ---: | ---: | ---: | --- |
| 8 GiB | 5.6 GiB | 0 | Passed, ~131 s per iteration |
| 6 GiB | 4.2 GiB | 2 | Passed after the restarts, ~363 s per iteration
(2.8x slower) |
| 5 GiB | 3.5 GiB | 2 (one per executor) | **No progress; stopped after 50
minutes** |
| 4 GiB | 2.8 GiB | 2 (one per executor) | **No progress; stopped after 50
minutes** |
At 5 and 4 GiB, each executor's container was OOM-killed once and restarted
by its Deployment (`restartCount` 1, `lastState.terminated.reason: OOMKilled`).
After that, the job neither completed nor failed: the runner was still waiting
when the sweep stopped it after ~3,000 s, against ~360 s for the whole 6 GiB
run including two iterations.
The executors' anonymous memory reached 3.7-4.3 GiB of the 4-5 GiB limits,
so the process outgrew the ~30% of the container outside the DataFusion memory
pool rather than the pool itself.
**To Reproduce**
With the cluster scripts in `benchmarks/cluster` (data on node-local disk in
our case; S3 should behave the same):
```sh
python3 run.py --engine ballista --query 18 --iterations 2 \
--executors 2 --executor-cpus 16 --vcores 16 --executor-memory-gib 5 \
--driver-cpus 4 --driver-memory-gib 16 --keep
```
Then watch the executor pods restart (`kubectl get pods -w`) and the runner
(`kubectl logs -f job/<run>-runner`) stop progressing.
Unfortunately the scheduler and executor logs of the two hung runs were not
captured before their pods were deleted, so the cause is not known yet. Things
worth checking:
- whether a restarted executor re-registers, and whether the scheduler
treats the old executor ID as lost (and reruns its shuffle outputs) or keeps
waiting for its running tasks;
- whether the restarted executor's in-flight tasks or vcore reservations are
ever released;
- whether the runner keeps polling a job whose stages can no longer be
scheduled.
**Expected behavior**
The job should recover as it did at 6 GiB (lost tasks and shuffle outputs
rerun), or fail with an error once retries are exhausted. It should not wait
indefinitely.
**Additional context**
The memory sweep's per-run records are in the memory usage page proposed in
#2855.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
