CurtHagenlocher commented on PR #504: URL: https://github.com/apache/arrow-dotnet/pull/504#issuecomment-6904514688
The latest review reported no open issues, but its summary mentioned "exception-safety leaks with caller-owned allocators" and "overlapping-memory corruption risk in bitmap appends" without locations. Following those up found real problems, all of which predate this PR: with the default allocator, finalizers reclaimed the memory, so the leaks only became permanent with caller-owned allocators. Two new commits fix them, each with tests that failed before the fix. **f8346b5: release IPC buffers when reading or writing fails** - **Writer:** all of a batch's temporary buffers (rewritten offsets, compression output) are allocated before writing, and each was released only after it was written. A stream that threw partway through (for example on a broken connection) leaked the rest, as did an exception while preparing the batch. The batch builder's `Dispose` now releases anything not yet released, and buffers are still released as soon as they're written. The test fails the stream at every write the writer makes, sync and async. - **Readers:** a message body was released only by handing it to a record batch, so a truncated stream, a failed decompression or a malformed message leaked it, as did a non-empty body on a schema or unrecognized message. `CreateArrowObjectFromMessage` now owns the body and releases it unless the result refers to it, and the stream readers release it if reading it fails. - **Bitmap appends:** appending bit by bit could overwrite source bits before reading them when the source was the builder's own memory and extended past its length. `BitmapBuilder` now copies a source that overlaps its memory whenever it appends it. **35c2e41: reference-count dictionary batch bodies** This one also leaked when nothing went wrong. A dictionary's arrays are views into its message body, but nothing owned the body, so every dictionary message read through a caller-owned allocator leaked it. Releasing it when the dictionary is replaced or the reader is disposed would be a use-after-free, because every batch using the dictionary shares it and can outlive both. So it now uses the same reference counting as shared buffers: - Dictionary buffers hold counted references to slices of the body, including buffers stored uncompressed in a compressed stream. - Each record batch retains its own reference to its dictionary (`ArrayData.Retain()`, no data copied), so disposing a batch releases only that batch's references. - The reader owns the dictionaries it keeps: it releases one when it's replaced, a delta and the dictionary it extends once they're merged, and all of them when the reader is disposed. Tests read consecutive dictionaries through a counting caller-owned allocator and check nothing is left outstanding, and check that both batches' dictionaries stay intact after the reader is disposed and the dictionary replaced, using an allocator that overwrites freed memory. The delta path isn't covered by a test, because the .NET writer can't produce delta dictionaries. One behaviour change with the default allocator: a dictionary body is now freed once the reader and every batch using it are disposed, rather than at finalization. Code that keeps using a dictionary after disposing both its batch and the reader would now read freed memory, which is the rule that already applies to a batch's own columns. All tests pass on net462, net472 and net8.0, including the pyarrow interop tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
