andygrove commented on code in PR #2845:
URL:
https://github.com/apache/datafusion-ballista/pull/2845#discussion_r4572916953
##########
ballista/core/src/execution_plans/buffer.rs:
##########
@@ -315,11 +316,15 @@ fn build_dam_stream(
// `get_array_memory_size()` ever changed between calls.
let mut buffered: VecDeque<(usize, RecordBatch)> = VecDeque::new();
let mut dam_broken = false;
+ // Counts each buffer once across the buffered batches, so zero-copy
+ // slices of one large array (as aggregates emit) are not charged
+ // once per slice.
+ let mut counter = RecordBatchMemoryCounter::new();
// Fill phase: read and buffer until pool pressure or end-of-stream.
while let Some(batch) = input.next().await {
let batch = batch?;
- let size = batch.get_array_memory_size();
+ let size = counter.count_batch(&batch);
Review Comment:
Good catch, thanks. Rebased onto #2718 and switched to
`RecordBatchMemoryTracker`: the drain now does
`res.shrink(tracker.uncount_batch(&batch))`, and the batch that breaks the dam
is uncounted right away since it never got reserved. The test now expects one
buffer's worth to stay reserved after each of the first nine slices and 0 after
the last (a0762a2ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]