rgbuilds commented on code in PR #27366:
URL: https://github.com/apache/datafusion/pull/27366#discussion_r4579345988


##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -1074,4 +1090,53 @@ mod tests {
             "expected a divergence internal error, got: {err}",
         );
     }
+
+    /// A batch whose clone shares every buffer with the original.
+    fn shared_buffer_batch() -> RecordBatch {
+        let schema = Arc::new(Schema::new(vec![Field::new("v", 
DataType::Int64, false)]));
+        RecordBatch::try_new(
+            schema,
+            vec![Arc::new(Int64Array::from_iter_values(0..1024))],
+        )
+        .unwrap()
+    }
+
+    #[test]
+    fn row_group_reverser_counts_shared_buffers_once() {
+        let batch = shared_buffer_batch();
+        let size = batch.get_array_memory_size();
+        let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(size + 
size / 2));
+        let reservation = MemoryConsumer::new("reverse-test").register(&pool);
+        let mut reverser = RowGroupReverser::new(None, Some(reservation));
+
+        // A clone shares every buffer with the original, so the second push
+        // must not charge the same buffers again.
+        reverser.push(batch.clone()).unwrap();
+        reverser.push(batch).unwrap();
+
+        assert!(

Review Comment:
   Please consider asserting the exact tracker-derived reservation and checking 
the intermediate reservation after the first pop. `<= size` plus final zero 
detects the old overcharge but would allow undercharging or releasing shared 
storage too early.



##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -817,22 +828,24 @@ impl RowGroupReverser {
             && let Some(front) = self.pending.front_mut()
         {
             let excess = self.pending_rows - limit;
-            *front = front.slice(excess, front.num_rows() - excess);
+            let original = front.clone();
+            let sliced = original.slice(excess, original.num_rows() - excess);
+            self.memory.count_batch_with_array_overhead(&sliced);
+            self.memory.uncount_batch_with_array_overhead(&original);
+            *front = sliced;
         }
         self.pending_rows = 0;
-        // The reversed copies replace the decoded batches, so the
-        // reservation is resized to exactly what is stored, which `pop`
-        // releases batch by batch.
+        // The reversed copies replace the decoded batches, so count each
+        // reversed batch before releasing the decoded one. Shared buffers stay
+        // charged until the last batch that holds them is released.
         for batch in self.pending.drain(..) {
-            self.reversed.push(reverse_batch_rows(&batch)?);
+            let reversed = reverse_batch_rows(&batch)?;
+            self.memory.count_batch_with_array_overhead(&reversed);
+            self.memory.uncount_batch_with_array_overhead(&batch);
+            self.reversed.push(reversed);
         }
         if let Some(reservation) = &self.reservation {

Review Comment:
   Could we make the resize-error state safe before returning it?
   
   With a batch plus its clone, pending memory is deduplicated enough for both 
pushes, but reversal creates two independent buffers. If `try_resize` then 
fails, the stream yields `Err` while retaining both reversed batches and the 
old, smaller reservation. Polling again pops one batch, exhausts the 
reservation, then panics on the second `shrink`.
   
   Please either make the error terminal or roll the 
retained/tracker/reservation state back consistently, and add this case as a 
regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to