rustyconover opened a new issue, #425:
URL: https://github.com/apache/arrow-dotnet/issues/425
# `ArrowStreamReader` hangs indefinitely reading a `RecordBatch` with a
zero-length body over a `NetworkStream`
## Summary
`ArrowStreamReaderImplementation.ReadMessageAsync` (and the sync
`ReadMessage`) reads a record
batch's body via `StreamExtensions.ReadFullBufferAsync`/`ReadFullBuffer`,
passing a
zero-length buffer whenever the batch has no buffers (e.g. a batch built
from a zero-column
schema). Over a `MemoryStream` this is a harmless no-op. Over a real
socket-backed
`NetworkStream`, a zero-byte `Stream.ReadAsync`/`Stream.Read` call does
**not** complete
immediately — it blocks as though waiting for the peer to send more data (or
close the
connection), instead of trivially returning `0`. In a lockstep/RPC-style
protocol where the
peer is itself waiting for a response before sending anything further, this
blocks forever.
## Repro
Minimal reproducer (no dependency on any RPC framework, just this package +
real `pyarrow`
over a Unix domain socket):
**Server (C#, stock `Apache.Arrow` NuGet package, no vendored/forked code):**
```csharp
using System.Net.Sockets;
using Apache.Arrow;
using Apache.Arrow.Ipc;
using Apache.Arrow.Types;
var sockPath = args[0];
if (File.Exists(sockPath)) File.Delete(sockPath);
using var listener = new Socket(AddressFamily.Unix, SocketType.Stream,
ProtocolType.Unspecified);
listener.Bind(new UnixDomainSocketEndPoint(sockPath));
listener.Listen();
Console.WriteLine("READY");
using var accepted = await listener.AcceptAsync();
var readStream = new NetworkStream(accepted, FileAccess.Read, ownsSocket:
false);
var writeStream = new NetworkStream(accepted, FileAccess.Write, ownsSocket:
false);
var outputSchema = new Schema([new Field("index", Int64Type.Default,
nullable: false)], null);
var tickSchema = new Schema([], null); // zero columns
using var writer = new ArrowStreamWriter(writeStream, outputSchema,
leaveOpen: true);
await writer.WriteStartAsync();
using var reader = new ArrowStreamReader(readStream, leaveOpen: true);
_ = await reader.GetSchema();
var sw = System.Diagnostics.Stopwatch.StartNew();
var tick = await reader.ReadNextRecordBatchAsync(); // <-- blocks for ~8s
(client's own timeout)
Console.WriteLine($"tick read in {sw.ElapsedMilliseconds}ms
(rows={tick?.Length ?? -1})");
```
**Client (Python, real `pyarrow`, matching how a typical socket-backed IPC
client is wired):**
```python
import pyarrow as pa
import pyarrow.ipc as ipc
import socket, sys
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect(sys.argv[1])
reader_stream = sock.makefile("rb")
writer_stream = sock.makefile("wb", buffering=0) # unbuffered — makes no
difference either way
tick_schema = pa.schema([]) # zero columns
tick_writer = ipc.new_stream(writer_stream, tick_schema)
output_reader = ipc.open_stream(reader_stream)
tick_batch = pa.record_batch([], schema=tick_schema)
tick_writer.write_batch(tick_batch) # writes real bytes synchronously —
confirmed via strace
# and via a separate BytesIO capture;
not the problem
response = output_reader.read_next_batch()
```
Result: the C# server's `ReadNextRecordBatchAsync()` call blocks for the
full client-side
socket timeout (tested at 5s, 10s, and 30s — it always takes *exactly* as
long as the client
is willing to wait, never less), even though the client wrote its complete
message bytes
synchronously and promptly. Once the client gives up and closes its socket,
the server's read
"completes" (returning `0` bytes, indistinguishable from an EOF), and the
batch is then
constructed correctly and instantly — the delay is 100% attributable to the
read call itself,
not any parsing or buffering elsewhere.
## Root cause (pinned via `strace`)
Running the server above under `strace -f -tt`, the sequence for the hung
read is:
```
recvmsg(fd, ..., iov_len=4) = 4 # continuation marker
recvmsg(fd, ..., iov_len=4) = 4 # message length = 64
recvmsg(fd, ..., iov_len=64) = 64 # full message body — arrives
correctly and promptly
recvmsg(fd, MSG_PEEK) = -1 EAGAIN
# ... nothing else happens on this fd for ~8 seconds ...
```
Adding timestamped instrumentation directly into
`ArrowStreamReaderImplementation.ReadMessageAsync`
and `StreamExtensions.ReadFullBufferAsync` confirms exactly where the time
goes: the 64-byte
*message* (the flatbuffer-encoded `Message` describing the batch) is read
correctly and
immediately. `message.BodyLength` for this batch is `0` (zero columns → zero
buffers → zero body
bytes needed). The subsequent call:
```csharp
IMemoryOwner<byte> bodyBuffOwner = AllocateMessageBodyBuffer(bodyLength);
// bodyLength == 0
Memory<byte> bodyBuff = bodyBuffOwner.Memory.Slice(0, bodyLength);
// 0-length slice
bytesRead = await BaseStream.ReadFullBufferAsync(bodyBuff,
cancellationToken)...
```
drills down into:
```csharp
// StreamExtensions.ReadFullBufferAsync
int bytesRead = await stream.ReadAsync(buffer.Slice(...),
cancellationToken)...
```
with `buffer.Length == 0`. This is the call that blocks for the full ~8
seconds. The
instrumented log shows it precisely:
```
DIAG-RFB [...746]: about to ReadAsync, want 0 more bytes (have 0/0)
DIAG-RFB [...689]: ReadAsync returned 0 bytes <- ~7.9 seconds later
```
Feeding the exact same captured wire bytes through a `MemoryStream` instead
of a real
`NetworkStream` parses correctly and instantly — `MemoryStream.ReadAsync`
with a zero-length
buffer returns `0` immediately, as one would expect. `NetworkStream` (backed
by a real
`Socket`) does not.
## Why this matters in practice
Any RPC/streaming protocol built on this library where a legitimate message
can have a
zero-length body (e.g. a "tick"/"continue" message carrying no payload, a
schema with no
columns, an empty control message) will hang the first time it's sent over a
real socket
transport, even though the writer sent everything correctly and promptly.
This is easy to miss
in testing because:
- `MemoryStream`-based tests never exercise the bug (zero-length reads are
trivially
synchronous there).
- Pipe-based transports may not exhibit it either (a hypothesis, not fully
confirmed) since
`FileStream`/anonymous-pipe reads for a zero-length buffer may behave
differently than a
`Socket`-backed `NetworkStream`.
- It only manifests over a real, separate-process, socket-backed connection.
## Suggested fix
`StreamExtensions.ReadFullBufferAsync` / `ReadFullBuffer` should
short-circuit
`buffer.Length == 0` and return `0` immediately, without ever calling
`stream.ReadAsync`/`Read`:
```csharp
public static async ValueTask<int> ReadFullBufferAsync(this Stream stream,
Memory<byte> buffer, CancellationToken cancellationToken = default)
{
if (buffer.Length == 0)
{
return 0;
}
int totalBytesRead = 0;
// ... unchanged ...
}
```
This matches the Go standard library's documented behavior for
`io.ReadFull`, which explicitly
returns immediately without issuing a read when `len(buf) == 0`, and
sidesteps whatever
`NetworkStream`/`Socket.ReceiveAsync`-specific semantics cause a zero-length
read to block on
readability rather than completing trivially.
We've applied this exact patch locally (in a vendored fork, since we also
carry an unrelated
custom_metadata patch) and confirmed it fixes the hang completely: 5
consecutive request/response
turns using zero-column batches now complete in single-digit milliseconds
over the same real
Unix-domain-socket setup that previously hung for the full duration of the
peer's timeout, every
time.
## Environment
- `Apache.Arrow` NuGet package version: 18.1.0 (also reproduces on the
version vendored from the
`v23.0.0` tag)
- .NET 9 and .NET 10, reproduced on Linux x86_64, Linux ARM64, macOS, and
Windows 11
- Reproduces regardless of whether the writer-side socket is buffered or
unbuffered
(`buffering=0`), and regardless of whether the reader uses split
read/write `NetworkStream`
instances or a single shared one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
