andygrove commented on code in PR #2845:
URL: 
https://github.com/apache/datafusion-ballista/pull/2845#discussion_r4572923807


##########
ballista/core/src/execution_plans/buffer.rs:
##########
@@ -315,11 +316,15 @@ fn build_dam_stream(
         // `get_array_memory_size()` ever changed between calls.
         let mut buffered: VecDeque<(usize, RecordBatch)> = VecDeque::new();
         let mut dam_broken = false;
+        // Counts each buffer once across the buffered batches, so zero-copy
+        // slices of one large array (as aggregates emit) are not charged
+        // once per slice.
+        let mut counter = RecordBatchMemoryCounter::new();
 
         // Fill phase: read and buffer until pool pressure or end-of-stream.
         while let Some(batch) = input.next().await {
             let batch = batch?;
-            let size = batch.get_array_memory_size();
+            let size = counter.count_batch(&batch);

Review Comment:
   Moving to draft until 2718 merges



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to