Siyao Meng created HDDS-17004:
---------------------------------
Summary: Streaming block read does not try another replica after a
client side checksum mismatch
Key: HDDS-17004
URL: https://issues.apache.org/jira/browse/HDDS-17004
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-fail-over-streaming-read-on-checksum-mismatch.patch,
TestBugCR1StreamReadChecksumNoFailover.java
h3. Mechanism
With {{ozone.client.verify.checksum=true}} (default)
{{StreamBlockInputStream.StreamingReader.onNext}} verifies each ReadBlock
response and fails the stream with {{OzoneChecksumException}} on a mismatch.
The reader surfaces it as {{IOException("Streaming read failed")}} wrapping an
{{ExecutionException}}. {{ConnectionFailureUtils.unwrapCause}} only stops at
{{StorageContainerException}} and {{TimeoutIOException}}, and
{{StreamBlockInputStream.handleExceptions}} (sequential reads) and
{{handlePreadException}} (positioned reads) retry only those and connectivity
errors. So the checksum error is rethrown and no other replica is tried. On the
sequential path the failed reader is also kept, so later sequential reads of
other ranges on the same stream fail with the same error until the stream is
unbuffered or reopened.
The non streaming path validates checksums inside the per datanode loop of
{{XceiverClientGrpc.sendCommandWithRetry}} and moves on to the next replica for
that request. Datanodes do not verify checksums for client reads by default, so
silent corruption on one replica reaches the client as successful responses
with bad bytes.
h3. Trigger
# {{ozone.client.stream.readblock.enable=true}}, a RATIS/THREE key.
# Silent corruption of the block data on one replica (one byte flipped in
place).
# Read the key with a client whose stream starts on that replica.
Which replica a stream starts on follows the order OM returns
({{ozone.network.topology.aware.read=true}} by default). Replicas at the same
distance are shuffled per lookup, so a client hits the corrupt replica first on
every read only when it is strictly the closest one (for example on the same
host), and otherwise on a share of the reads.
h3. Impact
* A streaming read that starts on the corrupt replica fails, although two
healthy replicas exist. The non streaming read of the same key succeeds.
* No wrong data is returned.
* Only with streaming block reads enabled (off by default).
* On the current code the failure appears as a false EOF ({{Inconsistent read
... numBytesRead=-1}}) instead of the checksum error, because of HDDS-16680:
the error arrives while {{StreamingReader.poll}} waits and is turned into end
of stream before the retry decision. The fix for this issue depends on
HDDS-16680 (apache/ozone PR #11388): only with both does the read succeed.
h3. Reproduction
PASS, deterministic in a mini cluster, unmodified source.
[^TestBugCR1StreamReadChecksumNoFailover.java]
({{hadoop-ozone/integration-test}}) writes a 32 KiB RATIS/THREE key on three
datanodes, flips one byte of the block file on the first pipeline node, then
reads the key 5 times without and 5 times with streaming block reads. All
datanodes have the client's address there, so topology aware read puts the
corrupted node first for every read. A passing test means the defect is
present. Results:
||Code||Non streaming||Streaming||Streaming failure||
|unmodified|5/5|0/5|false EOF|
|with the attached patch|5/5|0/5|false EOF (HDDS-16680)|
|with PR #11388 (HDDS-16680)|5/5|0/5|{{Streaming read failed}} caused by
{{OzoneChecksumException}}|
|with both|5/5|5/5|none, each read skips the corrupt replica once|
The short circuit read path appears to have the same gap ({{BlockInputStream}}
rereads the same local file after a checksum error until its retries are used
up); this is from code reading only, was not run and is not covered by the
attached patch.
h3. Patch
[^CR-1-fail-over-streaming-read-on-checksum-mismatch.patch], against
1fb64c8e499161d1452ae6948ffd461fc9d2c302. It also applies to master at
fd2f6092c6d, and together with PR #11388.
{{ConnectionFailureUtils.unwrapCause}} (used only by
{{StreamBlockInputStream}}) also returns an {{OzoneChecksumException}} found in
the cause chain. On such an error the sequential path ({{handleExceptions}})
and the positioned path ({{readPositioned}}) release the failed reader and skip
that replica, then open a stream on the next replica, as the non streaming path
does per request. The skip lasts for one positioned read, or for one refill of
the sequential read buffer (the set is cleared when the buffer is refilled).
This cannot loop: within that scope every checksum failure either adds a
replica not tried yet or rethrows, and a refill that succeeds returns data that
the read then consumes. A checksum mismatch is a fault of the data in one
range, not of the datanode, so the replica is not added to the stream's
{{failedStreamingDatanodes}}: reads of other ranges, and later reads of the
same range, start again from the first replica. There is no retry sleep, no
block location refresh and no use of the read retry limit
({{ozone.client.max.read.retries}}) for this case, on either path; it is
bounded by the number of replicas. When every replica fails the check, the read
throws {{IOException("Streaming read failed")}} with the
{{OzoneChecksumException}} of the last replica tried as its root cause (through
an {{ExecutionException}}), the same exception as today, and later reads of
other ranges still work.
Covered in the existing unit {{TestStreamBlockInputStream}} with three mocked
replicas, each case for sequential and positioned reads:
{{testChecksumMismatchFailsOverToAnotherReplica}} (first replica corrupt, the
read returns the data),
{{testChecksumMismatchOnAllReplicasSurfacesChecksumError}} (all replicas
corrupt, {{OzoneChecksumException}} is the root cause),
{{testChecksumMismatchInOneRangeDoesNotAffectOtherRanges}} (offset 0 corrupt on
all replicas, a read of offset 2 on the same stream returns the data) and
{{testChecksumSkipDoesNotUseReadRetry}} (first replica corrupt, second answers
CONTAINER_NOT_FOUND, one read retry allowed, the read returns the data from the
third). The responses arrive synchronously, so the tests do not depend on
HDDS-16680. Without the change the first fails with {{IOException: Streaming
read failed}} caused by {{OzoneChecksumException}}, and the third fails on its
sequential half because the failed reader is kept. With it
{{TestStreamBlockInputStream}} (33 tests), {{TestConnectionFailureUtils}} (21),
{{TestBlockInputStream}} (17), {{TestChunkInputStream}} (10),
{{TestMultipartInputStream}} (6), and in a mini cluster
{{TestStreamReadDatanodeFailover}} (3), {{TestStreamRead}} (4) and the
integration {{TestStreamBlockInputStream}} (3) pass, checkstyle is clean.
Related: HDDS-16873 proposes reporting replicas that fail client side checksum
verification; it assumes the client already moves to another replica.
Found by code review of the Ozone client read retry and failover paths, as part
of the TLA+ verification effort under HDDS-15926, on commit
1fb64c8e499161d1452ae6948ffd461fc9d2c302. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
