Richard-Durkee opened a new pull request, #2591:
URL: https://github.com/apache/datafusion-ballista/pull/2591
# Which issue does this PR close?
Related to #1426.
# Rationale for this change
When a task runs out of memory, the executor's error names only the
reservation that failed, for example `Failed to allocate additional 661.9 KB
for HashJoinInput[3] ...`. It doesn't say what else is holding the task's pool,
so you can't tell whether one operator is too big or several are sharing a
small budget.
DataFusion already solves this with `TrackConsumersPool`, which
`RuntimeEnvBuilder::with_memory_limit` uses by default. Ballista builds its
per-task pools itself, so it doesn't get it.
# What changes are included in this PR?
- Wrap each task's `FairSpillPool` in `TrackConsumersPool`, listing the top
five consumers, which matches DataFusion's default.
- Add a test that an exhausted-memory error names the largest consumers.
Example: TPC-H Q18 at SF10 with `prefer_hash_join=true`, 8 partitions and a
600MB executor pool.
Before:
```
Failed to allocate additional 661.9 KB for HashJoinInput[3] with 87.6 MB
already allocated for this reservation - 452.6 KB remain available for the
total memory pool: fair(pool_size: 572.2 MB)
```
After:
```
Additional allocation failed for HashJoinInput[3] with top memory consumers
(across reservations) as:
HashJoinInput[4]#40(can spill: false) consumed 100.0 MB, peak 100.0 MB,
HashJoinInput[5]#39(can spill: false) consumed 100.0 MB, peak 100.0 MB,
HashJoinInput[0]#45(can spill: false) consumed 100.0 MB, peak 100.0 MB,
HashJoinInput[1]#47(can spill: false) consumed 100.0 MB, peak 100.0 MB,
HashJoinInput[3]#49(can spill: false) consumed 87.6 MB, peak 87.6 MB.
Error: Failed to allocate additional 661.9 KB for HashJoinInput[3] ...
```
# Are there any user-facing changes?
Out-of-memory errors from executors with a memory pool configured now list
the task's largest memory consumers. There are no API or config changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
