Vivek1898 opened a new pull request, #27071: URL: https://github.com/apache/datafusion/pull/27071
## Which issue does this PR close? - Closes #27069. ## Rationale for this change Sorting batches that arrived over the Arrow C Data Interface as zero-copy slices of one larger batch panics on `branch-56` and `main`: ``` assertion failed: reservation.size() + shared.size() >= reserved_bytes ``` `RecordBatchMemoryCounter` identifies a buffer by its start address and adds the capacity of the first buffer it sees at that address. That is right for ordinary slices, which all report the capacity of the allocation they share. A buffer built by `Buffer::from_custom_allocation`, which is what `arrow::ffi::from_ffi` produces, reports the length it was imported with instead, so slices of one allocation share an address but disagree on size. When the sort's runs share buffers, the shared input reservation is built from the counter and can end up smaller than a later run's own count, and `sort_run` asserts on exactly that. ## What changes are included in this PR? `BufferIdSet` now remembers the largest capacity seen for each address, inline and after promotion to the hash map, and `insert` returns the bytes that were not counted yet: the whole capacity for a new address, the growth when a larger capacity shows up for a known one, and zero otherwise. The counter adds whatever `insert` returns. For buffers that agree on capacity, which is every non-FFI case, nothing changes. `RecordBatchMemoryTracker` keys buffers the same way and subtracts the capacity of whichever buffer is uncounted. I have not changed it here; if a reviewer thinks the same mismatch should be handled there, I can follow up, but I did not want to widen this fix past the sort path from the issue. ## What is the testing strategy for this PR? Two unit tests in `memory.rs` build two `Int32` arrays over one allocation through `Buffer::from_custom_allocation`, declaring 16 and 1024 values, the shape `from_ffi` produces: - `test_record_batch_memory_counter_takes_largest_capacity_for_shared_address` counts them in both orders and checks the total is the larger capacity either way. - `test_record_batch_memory_counter_takes_largest_capacity_after_promotion` does the same after the inline slots have been promoted to the hash map. The existing shared-buffer and promotion tests still pass unchanged. ## Are there any user-facing changes? No. Memory accounting for FFI-imported slices is now at least as large as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
