[
https://issues.apache.org/jira/browse/HDDS-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16680:
----------------------------------
Labels: pull-request-available (was: )
> StreamBlockInputStream returns EOF instead of failing over when an error
> response arrives during poll()
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16680
> URL: https://issues.apache.org/jira/browse/HDDS-16680
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.1
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
> Labels: pull-request-available
>
> A streaming read that gets an error response from a datanode (e.g.
> UNKNOWN_BCSID, CONTAINER_NOT_FOUND, or a token failure) can return EOF
> instead of throwing. Retry and failover to another replica are skipped, and
> KeyInputStream fails with:
> {code:java}
> java.io.IOException: Inconsistent read for blockID=conID: 1 locID:
> 117883640217600001 bcsId: 2 replicaIndex: null length=32768 position=0
> numBytesToRead=32768 numBytesRead=-1
> {code}
> Root cause: StreamingReader.poll() runs checkError() at the top of its loop,
> then waits up to 100 ms on the response queue, then returns null (end of
> stream) if future.isDone(). If onNext() (via failOnErrorResponse()) or
> onError() fails the future during that wait, poll() returns null for a
> stream that failed. The error is lost, and read() turns the null into EOF.
> Regression: two changes combined to cause this:
> - HDDS-15479 moved the isDone() check to after the queue wait (to avoid
> dropping the last queued item). That put the whole wait between checkError()
> and isDone().
> - HDDS-16207 made error responses and gRPC errors other than CANCELLED fail
> the future right away. Before that, they ended in the read timeout, and
> TimeoutIOException triggered a retry/failover (HDDS-15521). After HDDS-16207,
> the failure almost always lands during the wait.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
