adamreeve opened a new issue, #40517:
URL: https://github.com/apache/arrow/issues/40517
### Describe the enhancement requested
Currently the C# `ArrowStreamWriter` always writes all data in a buffer.
This behaviour differs to the Python/C++ implementation, which only writes
slices of the buffers when an array has a nonzero offset or size in bytes less
than the buffer length. This can be observed by looking at the sizes of IPC
files for a whole RecordBatch, compared to slices of the data.
Python:
```python
import pyarrow as pa
import numpy as np
num_rows = 400
rows_per_batch = 100
ints = pa.array(np.arange(0, num_rows, 1, dtype=np.int32))
floats = pa.array(np.arange(0, num_rows / 10.0, 0.1, dtype=np.float32))
all_data = pa.RecordBatch.from_arrays([ints, floats], names=["a", "b"])
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, all_data.schema) as writer:
writer.write_batch(all_data)
buf = sink.getvalue()
print(f"Size of serialized full batch = {buf.size}")
for offset in range(0, num_rows, rows_per_batch):
slice = all_data.slice(offset, rows_per_batch)
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, slice.schema) as writer:
writer.write_batch(slice)
buf = sink.getvalue()
print(f"Size of serialized slice at offset {offset} = {buf.size}")
```
This outputs:
```
Size of serialized full batch = 3576
Size of serialized slice at offset 0 = 1176
Size of serialized slice at offset 100 = 1176
Size of serialized slice at offset 200 = 1176
Size of serialized slice at offset 300 = 1176
```
The size of the full batch is 1/4 the full batch after accounting for the
overhead of metadata.
Doing the same in C#:
```C#
const int numRows = 400;
const int rowsPerBatch = 100;
var allData = new RecordBatch.Builder()
.Append("a", false, col => col.Int32(array =>
array.AppendRange(Enumerable.Range(0, numRows))))
.Append("b", false, col => col.Float(array =>
array.AppendRange(Enumerable.Range(0, numRows).Select(i => 0.1f * i))))
.Build();
{
using var ms = new MemoryStream();
using var writer = new ArrowFileWriter(ms, allData.Schema, false, new
IpcOptions());
await writer.WriteStartAsync();
await writer.WriteRecordBatchAsync(allData);
await writer.WriteEndAsync();
Console.WriteLine($"Size of serialized full batch = {ms.Length}");
}
for (var offset = 0; offset < allData.Length; offset += rowsPerBatch)
{
var arraySlices = allData.Arrays
.Select(arr => ArrowArrayFactory.Slice(arr, offset, rowsPerBatch))
.ToArray();
var slice = new RecordBatch(allData.Schema, arraySlices,
arraySlices[0].Length);
using var ms = new MemoryStream();
using var writer = new ArrowFileWriter(ms, slice.Schema, false, new
IpcOptions());
await writer.WriteStartAsync();
await writer.WriteRecordBatchAsync(slice);
await writer.WriteEndAsync();
Console.WriteLine($"Size of serialized slice at offset {offset} =
{ms.Length}");
}
```
This outputs:
```
Size of serialized full batch = 3802
Size of serialized slice at offset 0 = 3802
Size of serialized slice at offset 100 = 3802
Size of serialized slice at offset 200 = 3802
Size of serialized slice at offset 300 = 3802
```
Writing a slice of the data results in the same file size as writing the
full data, but we'd like to be able to break IPC data into smaller slices in
order to send it over a transport that has a message size limit.
From a quick look at the C++ implementation, one complication is dealing
with null bitmaps, which need to be copied to ensure the start is aligned with
a byte boundary.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
