Sergey Soldatov created HDDS-16207:
--------------------------------------
Summary: Streaming read hangs until read timeout on out-of-range
ReadBlock offsets
Key: HDDS-16207
URL: https://issues.apache.org/jira/browse/HDDS-16207
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client, Ozone Datanode
Affects Versions: 2.3.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
When a streaming read (StreamBlockInputStream / ReadBlock) requests an offset
at or past the end of the block, the client does not get an error or an EOF —
it blocks until the stream read timeout (default PT10S) expires. Three defects
combine to cause this:
1. Client pre-read overshoot. StreamBlockInputStream.readBlock() adds the
pre-read size (up to 32MB) to the required length without clamping against
blockLength, so requestedLength can run past the physical end of the block. The
next pre-read request then goes out with an offset beyond EOF — on a perfectly
normal read near the end of a block.
2. Datanode silent failure. For such a request, KeyValueHandler.readBlockImpl
computes requiredLength <= 0 and simply falls through the read loop: no data
response, no error, no half-close. The stream stays open and idle. A misaligned
variant is also possible: an offset past EOF whose checksum-aligned floor is
still inside the block streams back data that ends before the requested offset,
which leaves the client waiting for bytes that never arrive.
3. Client drops streamed errors. Even when the datanode does report a
problem, StreamingReader ignores it: onError() only handles CANCELLED
StatusRuntimeExceptions (any other gRPC status is silently discarded without
failing the stream), and onNext() reads the payload without checking the
response result, so an error response proto (e.g. CONTAINER_NOT_FOUND, token
verification failure) is treated as an empty data message.
In every case the pending poll() on the response queue just times out,
surfacing as a 10-second stall / TimeoutIOException instead of an immediate
error.
This was discovered during testing of HDDS-15422 using YCSB HBase workload.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
