buraksenn opened a new pull request, #24427:
URL: https://github.com/apache/datafusion/pull/24427
## Which issue does this PR close?
- Part of #23385.
## Rationale for this change
LimitedBatchCoalescer retains in-progress Arrow buffers and completed
batches waiting to be polled, but this memory was not charged to DataFusion's
memory pool.
FilterExec, CoalesceBatchesExec, HashJoinExec, AsyncFuncExec, and
RepartitionExec could underreport retained memory suicne they use
LimitedBatchCoalescer.
After https://github.com/apache/arrow-rs/pull/10331, which added
BatchCoalescer::size() by @rluvaton we can account for this memory.
## What changes are included in this PR?
Adds reservation-backed memory accounting to LimitedBatchCoalescer and its
existing callers. Repartition can flush partial batches through its normal
send/spill path under pressure. The old untracked constructor remains available
but is deprecated.
Arrow reports coalescer size after mutation, so already-allocated memory
remains charged while an error propagates. Shared or sliced arrays may be
conservatively charged for their full backing buffers.
Remaining operators that use Arrow's BatchCoalescer directly will be
handled as follow-up work for #23385.
## Are these changes tested?
Yes mostly unit tests in LimitedBatchCoalescer
## Are there any user-facing changes?
`LimitedBatchCoalescer:;new` method is deprecated and `new_with_reservation`
method is added. Additionally, there can be errors due to memory limitations
such as `ResourcesExhausted`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
