ranflarion commented on PR #23032: URL: https://github.com/apache/datafusion/pull/23032#issuecomment-5478277944
We hit the same failure modes in a setup similar to DataFusion Comet (2x build-side peak from concat_batches, and the all-or-nothing single allocation) and ended up with an NLJ-scoped version of this change that we'd like to upstream: the build side is kept as target-sized chunks produced by arrow's BatchCoalescer, with with_biggest_coalesce_batch_size retaining already-large batches zero-copy, the visited bitmap indexed by global row numbers over prefix-sum chunk offsets, and the spill write path going through the same coalescer so the memory-limited replay reads back uniform chunks. On an 880MB build side (20M rows, inequality join, target_partitions=1), pristine main peaks at 1738MB RSS vs 899MB with chunks, and with --memory-limit 1g main completes while peaking at 1739MB because the concat output is never reserved in the pool. @mbutrovich the two scenarios you raised earlier measure as follows on that branch: a build side arriving as 250,000 8-row batches probes at parity with main (7.36s vs 7.20s medians over interleaved runs, ~1.6e10 pair evaluations) because the coalescer compacts to target size before probing ever starts, and sliced batches below half the target size are compacted the same way, while larger ones are retained zero-copy with the reservation charging get_array_memory_size of the full parent buffers, which is conservative; precise dedup-by-allocation accounting is a planned follow-up. Issue at #24819 and PR at #24820 extract just this operator, leaving hash join and piecewise merge to this PR or follow-ups — @maxburke happy to coordinate if a stepping-stone is useful here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
