andygrove opened a new issue, #2321:
URL: https://github.com/apache/datafusion-ballista/issues/2321
**Describe the bug**
TPC-H Q10 at SF1000 fails under the AQE plan produced after #2315, hitting a
memory-pool OOM in `SortPreservingMergeExec`. On the pre-#2315 baseline Q10
completed in ~53s on the same cluster and config; after #2315 it errors at
runtime.
Full error:
```
DataFusion error: Arrow error: External error: Execution error:
Job 0380C6NW0A000 failed: Job failed due to stage 6 failed:
Task failed due to runtime execution error:
DataFusionError(Shared(External(DataFusionError(ResourcesExhausted(
"Failed to allocate additional 111.2 MB for SortPreservingMergeExec[0]
with 5.5 GB already allocated for this reservation
- 19.4 MB remain available for the total memory pool:
fair(pool_size: 5.6 GB)"
)))))
```
**To Reproduce**
- Ballista built from `f21c958` (merge commit of #2315)
- TPC-H SF1000, parquet/zstd, on Kubernetes: 32 executors × 8 concurrent
tasks
- Memory pool: `fair` pool sized at 70% of a 64 GiB executor container ≈
44.8 GB total, divided across 8 concurrent tasks → ~5.6 GB per-task budget
(matches the error trace)
- AQE default-on (this PR flips it)
**Expected behavior**
Q10 should complete without OOM at this scale. It passed at ~53s on the same
infrastructure before #2315.
**Additional context**
- Full 22-query run: 19 passed, Q10 regressed as above, Q21/Q22 hit
unrelated cluster networking issues.
- Q11 was fixed by this PR (previously erroring, now 15.2s).
- Overall wall-clock improved ~12% (632s vs 721s baseline) despite the Q10
regression.
- The per-task pool sizing didn't change on our side — the new AQE plan for
Q10 evidently needs more sort/merge memory than the previous plan. Possible
causes: the removed redundant shuffle changing partition sizes, or an
AQE-driven join/coalesce decision concentrating data.
- Original benchmark report:
https://github.com/apache/datafusion-ballista/pull/2315#issuecomment-5308644923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
