Sergey Soldatov created HDDS-17873:
--------------------------------------
Summary: StreamBlockInputStream should cancel, not half-close, a
failed streaming read
Key: HDDS-17873
URL: https://issues.apache.org/jira/browse/HDDS-17873
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 2.3.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
When a streaming read fails, the client fails over to another replica. A
common case is a stalled datanode that runs past
ozone.client.stream.read.timeout. The client then closes the old stream the
same way it closes a successful one: with a gRPC half-close (onCompleted()). A
half-close does not end the call. It only tells the datanode that no more
requests are coming.
As a result:
- The datanode handles one call's requests in order, so it sees the
half-close only after the stuck request finishes. It then still sends the rest
of that request's data to a client that has already moved on. With the default
pre-read size, that can be about 32 MB.
- On the client, the abandoned call and its reader stay alive until then, and
the late data is queued in the abandoned reader. The streaming call has no gRPC
deadline, so if the stuck request never finishes, nothing ends the call.
- Each failover logs "Failed to setCompleted: already completed
exceptionally" twice, with stack traces.
Positioned reads (pread) close a failed attempt the same way.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]