CurtHagenlocher commented on PR #504:
URL: https://github.com/apache/arrow-dotnet/pull/504#issuecomment-6904514688

   The latest review reported no open issues, but its summary mentioned 
"exception-safety leaks with caller-owned allocators" and "overlapping-memory 
corruption risk in bitmap appends" without locations. Following those up found 
real problems, all of which predate this PR: with the default allocator, 
finalizers reclaimed the memory, so the leaks only became permanent with 
caller-owned allocators. Two new commits fix them, each with tests that failed 
before the fix.
   
   **f8346b5: release IPC buffers when reading or writing fails**
   - **Writer:** all of a batch's temporary buffers (rewritten offsets, 
compression output) are allocated before writing, and each was released only 
after it was written. A stream that threw partway through (for example on a 
broken connection) leaked the rest, as did an exception while preparing the 
batch. The batch builder's `Dispose` now releases anything not yet released, 
and buffers are still released as soon as they're written. The test fails the 
stream at every write the writer makes, sync and async.
   - **Readers:** a message body was released only by handing it to a record 
batch, so a truncated stream, a failed decompression or a malformed message 
leaked it, as did a non-empty body on a schema or unrecognized message. 
`CreateArrowObjectFromMessage` now owns the body and releases it unless the 
result refers to it, and the stream readers release it if reading it fails.
   - **Bitmap appends:** appending bit by bit could overwrite source bits 
before reading them when the source was the builder's own memory and extended 
past its length. `BitmapBuilder` now copies a source that overlaps its memory 
whenever it appends it.
   
   **35c2e41: reference-count dictionary batch bodies**
   
   This one also leaked when nothing went wrong. A dictionary's arrays are 
views into its message body, but nothing owned the body, so every dictionary 
message read through a caller-owned allocator leaked it. Releasing it when the 
dictionary is replaced or the reader is disposed would be a use-after-free, 
because every batch using the dictionary shares it and can outlive both. So it 
now uses the same reference counting as shared buffers:
   - Dictionary buffers hold counted references to slices of the body, 
including buffers stored uncompressed in a compressed stream.
   - Each record batch retains its own reference to its dictionary 
(`ArrayData.Retain()`, no data copied), so disposing a batch releases only that 
batch's references.
   - The reader owns the dictionaries it keeps: it releases one when it's 
replaced, a delta and the dictionary it extends once they're merged, and all of 
them when the reader is disposed.
   
   Tests read consecutive dictionaries through a counting caller-owned 
allocator and check nothing is left outstanding, and check that both batches' 
dictionaries stay intact after the reader is disposed and the dictionary 
replaced, using an allocator that overwrites freed memory. The delta path isn't 
covered by a test, because the .NET writer can't produce delta dictionaries.
   
   One behaviour change with the default allocator: a dictionary body is now 
freed once the reader and every batch using it are disposed, rather than at 
finalization. Code that keeps using a dictionary after disposing both its batch 
and the reader would now read freed memory, which is the rule that already 
applies to a batch's own columns.
   
   All tests pass on net462, net472 and net8.0, including the pyarrow interop 
tests.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to