adriangb commented on PR #23390: URL: https://github.com/apache/datafusion/pull/23390#issuecomment-4922305113
## `large_values` re-run on the final design (head 16a8de0a16) — plus a bug it caught Re-running the full matrix surfaced a real bug in the byte-target path, now fixed (commit 16a8de0a16): arrow's `take`/`interleave` copy view-array *views* but **share the data buffers**, so byte-targeted sorted chunks and merged batches pinned every source batch they drew from — a 4MB-logical merged batch retained 128MB (`SortExec` reported `output_bytes=31.9GB` for 1GB of data), and since `BatchBuilder`/`SortPreservingMergeExec` reservations count retained size, `ORDER BY <string>` **failed at a 4GB limit where row-only chunking passed**. Fix: `gc_view_arrays` on `IncrementalSortIterator` chunks and `BatchBuilder` output when a byte target is set (heuristic-gated, no-ops for non-view/non-wasteful batches; row-only mode untouched). Two red-first regression tests added. Final matrix (16KiB values = 1GiB dataset unless noted; `off` = `adaptive_target_batch_size=false`, `auto` = default; interleaved runs, quiet machine, AC power): | scenario | off | auto (default) | |---|---|---| | unlimited, all 5 queries x2 rounds | baseline | identical within ~1% (provably same code path) | | 2G n=4 | Q2/Q3 FAIL | **all 5 pass** (Q2 488ms, Q3 506ms); passing queries +3-6% | | 2G n=12 | Q2/Q3 FAIL | **all 5 pass** (Q2 367ms, Q3 500ms — the former Q3 flake is gone post-fix) | | 4G n=12 | all pass | all pass; +10-39% on this worst-case dataset (adaptive still splits 128MB batches while `off` fits comfortably — the insurance premium when a finite limit is set; `adaptive_target_batch_size=false` opts out) | | 64KiB values (2GiB dataset), 2G, n=4 / n=12 | Q2 FAIL | **Q2 passes both** (~1.1s, down from ~2.3s before the view-GC fix) | 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UcPTREZVLXsSZDRCae33gm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
