vinycalves commented on code in PR #27366:
URL: https://github.com/apache/datafusion/pull/27366#discussion_r4598307855


##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -817,22 +828,24 @@ impl RowGroupReverser {
             && let Some(front) = self.pending.front_mut()
         {
             let excess = self.pending_rows - limit;
-            *front = front.slice(excess, front.num_rows() - excess);
+            let original = front.clone();
+            let sliced = original.slice(excess, original.num_rows() - excess);
+            self.memory.count_batch_with_array_overhead(&sliced);
+            self.memory.uncount_batch_with_array_overhead(&original);
+            *front = sliced;
         }
         self.pending_rows = 0;
-        // The reversed copies replace the decoded batches, so the
-        // reservation is resized to exactly what is stored, which `pop`
-        // releases batch by batch.
+        // The reversed copies replace the decoded batches, so count each
+        // reversed batch before releasing the decoded one. Shared buffers stay
+        // charged until the last batch that holds them is released.
         for batch in self.pending.drain(..) {
-            self.reversed.push(reverse_batch_rows(&batch)?);
+            let reversed = reverse_batch_rows(&batch)?;
+            self.memory.count_batch_with_array_overhead(&reversed);
+            self.memory.uncount_batch_with_array_overhead(&batch);
+            self.reversed.push(reversed);
         }
         if let Some(reservation) = &self.reservation {

Review Comment:
   Made the failure terminal and rolled back the state: finish now reverses 
into a local vec first; on reversal or resize failure it uncounts all retained 
batches, resets the reservation to zero, and sets a failed flag so pop returns 
None and push/finish error. Added a regression that fails the resize and polls 
again.



##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -1074,4 +1090,53 @@ mod tests {
             "expected a divergence internal error, got: {err}",
         );
     }
+
+    /// A batch whose clone shares every buffer with the original.
+    fn shared_buffer_batch() -> RecordBatch {
+        let schema = Arc::new(Schema::new(vec![Field::new("v", 
DataType::Int64, false)]));
+        RecordBatch::try_new(
+            schema,
+            vec![Arc::new(Int64Array::from_iter_values(0..1024))],
+        )
+        .unwrap()
+    }
+
+    #[test]
+    fn row_group_reverser_counts_shared_buffers_once() {
+        let batch = shared_buffer_batch();
+        let size = batch.get_array_memory_size();
+        let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(size + 
size / 2));
+        let reservation = MemoryConsumer::new("reverse-test").register(&pool);
+        let mut reverser = RowGroupReverser::new(None, Some(reservation));
+
+        // A clone shares every buffer with the original, so the second push
+        // must not charge the same buffers again.
+        reverser.push(batch.clone()).unwrap();
+        reverser.push(batch).unwrap();
+
+        assert!(

Review Comment:
   Strengthened both tests: the push test asserts the exact tracker-derived 
reservation, and the release test checks the intermediate reservation after the 
first pop using the first batch's tracker size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to