Copilot commented on code in PR #504:
URL: https://github.com/apache/arrow-dotnet/pull/504#discussion_r4581130062
##########
src/Apache.Arrow/Ipc/ArrowReaderImplementation.cs:
##########
@@ -230,27 +294,39 @@ private List<IArrowArray> BuildArrays(
return arrays;
}
- using var bufferCreator =
GetBufferCreator(recordBatchMessage.Compression);
- var recordBatchEnumerator = new RecordBatchEnumerator(in
recordBatchMessage);
- int schemaFieldIndex = 0;
- do
+ try
{
- Field field = schema.GetFieldByIndex(schemaFieldIndex++);
- Flatbuf.FieldNode fieldNode =
recordBatchEnumerator.CurrentNode;
+ using var bufferCreator =
GetBufferCreator(recordBatchMessage.Compression, body);
+ var recordBatchEnumerator = new RecordBatchEnumerator(in
recordBatchMessage);
+ int schemaFieldIndex = 0;
+ do
+ {
+ Field field = schema.GetFieldByIndex(schemaFieldIndex++);
+ Flatbuf.FieldNode fieldNode =
recordBatchEnumerator.CurrentNode;
- ArrayData arrayData = LoadField(version, ref
recordBatchEnumerator, field, in fieldNode, messageBuffer, bufferCreator);
+ ArrayData arrayData = LoadField(version, ref
recordBatchEnumerator, field, in fieldNode, messageBuffer, bufferCreator);
- arrays.Add(ArrowArrayFactory.BuildArray(arrayData));
- } while (recordBatchEnumerator.MoveNextNode());
+ arrays.Add(ArrowArrayFactory.BuildArray(arrayData));
Review Comment:
If `LoadField` has already created one or more buffers in `arrowBuff` and
then parsing a later buffer or child throws, the `BuildArrays` catch only
disposes arrays that were already added to `arrays`; the partially built
`ArrayData` was never added, so those buffers are leaked. This is observable
with a caller-owned allocator (for example, malformed IPC after a decompressed
buffer was allocated), and the new allocator path makes the leak permanent.
Track and dispose the in-progress buffers/children when `LoadField` fails, or
transfer ownership only after the whole field has been built.
##########
src/Apache.Arrow.Operations/Shredding/ShreddedVariantArrayBuilder.cs:
##########
@@ -83,7 +83,7 @@ public static VariantArray Build(
// metadata column: emit the shared bytes once per row. (A
dictionary-encoded
// or run-end-encoded representation would compress this;
VariantArray's reader
// already handles those, but for simplicity we emit the plain
binary form.)
- BinaryArray.Builder metadataBuilder = new BinaryArray.Builder();
+ using BinaryArray.Builder metadataBuilder = new
BinaryArray.Builder(BinaryType.Default, allocator);
Review Comment:
The top-level validity bitmap is still constructed with `new
ArrowBuffer.BitmapBuilder(rowCount)` rather than the supplied `allocator`, so
an explicit allocator causes this bitmap to grow in managed scratch memory and
then be copied during `validity.Build(allocator)`. That leaves this shredding
path with an extra allocation/copy, unlike the allocator-backed child builders
added below; construct the validity builder with the same allocator so the
output can be handed over directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]