adriangb commented on PR #23390:
URL: https://github.com/apache/datafusion/pull/23390#issuecomment-4920276998
## Adaptive byte target: derived from the memory limit (commit 775c059d88)
Per review discussion, the byte target can now derive itself instead of
requiring manual configuration. New option
`datafusion.execution.adaptive_target_batch_size` (default **true**):
- Explicit `target_batch_size_bytes` always wins.
- Otherwise, if the memory pool reports a **finite** limit: `target =
min(pool_size / target_partitions / 16, 16MiB)`. The `/16` margin covers the
per-batch multipliers in the sort/merge machinery (~2x sort buffering, ~4x
spill-merge reservation per stream with read-ahead, SPM buffering every
stream); the 16MiB ceiling is where tight-pool merges stay reliably stable and
beyond which bigger batches add no amortization.
- **No memory limit → no target**: the normalizer wrapper is not installed
at all, so the unlimited case is the identical code path to today ("cases that
pass never get slower" holds by construction).
- Derived target below 1MiB → adaptive stays off: a pool that small cannot
be saved by re-chunking, and carefully crafted small-pool OOM scenarios (e.g.
the `memory_limit` test suite) keep their exact behavior.
Acceptance on `large_values` with **zero configuration** (just
`--memory-limit 2G`):
| scenario | before | adaptive |
|---|---|---|
| unlimited memory | baseline | identical code path |
| 16KiB values (1GiB), 2G, n=4 | Q2/Q3 FAIL | all 5 queries pass |
| 16KiB values, 2G, n=12 | Q2/Q3 FAIL | pass (rare Q3 flake at this 2:1
data:pool boundary) |
| 64KiB values (2GiB data), 2G, n=4 / n=12 | FAIL | both pass |
Trade-off note: with a finite limit, already-passing queries whose batches
exceed the derived threshold pay the ~5-10% compaction tax (a scan-only query
doesn't know whether a downstream sort needs protection). Full workspace test
suite passes, including the memory-limit OOM-assertion tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UcPTREZVLXsSZDRCae33gm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
