Siyao Meng created HDDS-16781:
---------------------------------
Summary: Intermittent failure in TestKeyInputStream#testSkip
Key: HDDS-16781
URL: https://issues.apache.org/jira/browse/HDDS-16781
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Client, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR]
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
-- Time elapsed: 22.70 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
{code}
h3. Root cause
{{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
the data is committed, but a follower may not have applied the PutBlock yet. In
the failing run both ReadChunk requests first went to such a follower:
{code}
WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to find
the block with bcsID 485. Container 9 bcsId is 483. , Result: UNKNOWN_BCSID
{code}
The client correctly retried on another datanode, but the metric counts every
attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
h3. Fix
Close the key's container and wait until it is CLOSED on all replicas before
reading. Followers apply the close only after the PutBlock (Ratis log order),
so no replica can lag behind the block's bcsId.
- https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
