[
https://issues.apache.org/jira/browse/HDDS-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16781:
------------------------------
Description:
{code}
[ERROR]
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
-- Time elapsed: 22.70 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
{code}
h3. Root cause
{{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
the data is committed, but a follower may not have applied the PutBlock yet. In
the failing run both ReadChunk requests first went to such a follower:
{code}
WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to find
the block with bcsID 485. Container 9 bcsId is 483. , Result: UNKNOWN_BCSID
{code}
The client correctly retried on another datanode, but the metric counts every
attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
h3. Fix
Close the key's container and wait until it is CLOSED on all replicas before
reading. Followers apply the close only after the PutBlock (Ratis log order),
so no replica can lag behind the block's bcsId. {{testSeek}} has the same exact
ReadChunk count assertion after a write, so the wait is applied there too.
- https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
was:
{code}
[ERROR]
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
-- Time elapsed: 22.70 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
{code}
h3. Root cause
{{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
the data is committed, but a follower may not have applied the PutBlock yet. In
the failing run both ReadChunk requests first went to such a follower:
{code}
WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to find
the block with bcsID 485. Container 9 bcsId is 483. , Result: UNKNOWN_BCSID
{code}
The client correctly retried on another datanode, but the metric counts every
attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
h3. Fix
Close the key's container and wait until it is CLOSED on all replicas before
reading. Followers apply the close only after the PutBlock (Ratis log order),
so no replica can lag behind the block's bcsId.
- https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
> Intermittent failure in TestKeyInputStream#testSkip
> ---------------------------------------------------
>
> Key: HDDS-16781
> URL: https://issues.apache.org/jira/browse/HDDS-16781
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR]
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
> -- Time elapsed: 22.70 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
> {code}
> h3. Root cause
> {{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
> exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
> the data is committed, but a follower may not have applied the PutBlock yet.
> In the failing run both ReadChunk requests first went to such a follower:
> {code}
> WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to
> find the block with bcsID 485. Container 9 bcsId is 483. , Result:
> UNKNOWN_BCSID
> {code}
> The client correctly retried on another datanode, but the metric counts every
> attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
> h3. Fix
> Close the key's container and wait until it is CLOSED on all replicas before
> reading. Followers apply the close only after the PutBlock (Ratis log order),
> so no replica can lag behind the block's bcsId. {{testSeek}} has the same
> exact ReadChunk count assertion after a write, so the wait is applied there
> too.
> - https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
