pmoust opened a new issue, #26166: URL: https://github.com/apache/datafusion/issues/26166
### Describe the bug `RepartitionExec`'s output coalescer holds in-progress buffers that are never charged to any `MemoryReservation`. Only completed batches are charged. In DataFusion 54.1.0 (`datafusion-physical-plan/src/repartition/mod.rs`): - `OutputChannel::coalesce` (around lines 168-172) pushes every hash-partitioned batch into a `SharedCoalescer` (`push_and_drain`, around lines 244-252). - The coalescer copies string views into new buffers. See arrow-select 58.4.0, `coalesce/byte_view.rs`: `next_buffer` (around line 384) and `ensure_capacity` (around line 102); and `coalesce/primitive.rs` (around line 59). - `send` charges only a completed batch: `get_array_memory_size` at around line 181, then `try_grow` at around line 186. The receiver releases that charge when it takes the batch (around line 1873). - Nothing charges the coalescer's in-progress buffers. ### To Reproduce A heap profile taken during ClickBench Q35 (string-keyed `GROUP BY` over 20M rows, 32 partitions) attributed about 106 MB of live heap to these coalescer buffers. Those bytes were outside every reservation. At that moment that was about 26% of the gap between allocated memory and the SQL pool's reservation. The share scales with partition count and `batch_size`. ### Expected behavior The output channel's reservation covers the bytes the coalescer has buffered but not yet emitted, so a `MemoryPool` limit bounds them. ### Additional context Lowering `datafusion.execution.batch_size` or `target_partitions` bounds the buffered size but does not make it visible to the pool. Related: #23393. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
