[
https://issues.apache.org/jira/browse/HDDS-16967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128023#comment-18128023
]
Siyao Meng commented on HDDS-16967:
-----------------------------------
Same lagging follower, third symptom, in {{testBlockWritesWithDnFailures}}:
{code}
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testBlockWritesWithDnFailures
<<< ERROR!
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
ContainerID 6 does not exist
at
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testBlockWritesWithDnFailures(TestFailureHandlingByClient.java:215)
{code}
The client committed the first block on a majority only, and the test shut down
{{datanodes.get(0)}} and {{get(1)}}, which were that majority. The PR now keeps
the Ratis leader running and shuts down the two followers.
- https://github.com/apache/ozone/actions/runs/38924347415/job/117067399755
> Intermittent failure in
> TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16967
> URL: https://issues.apache.org/jira/browse/HDDS-16967
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 295.5 s <<<
> FAILURE! -- in org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
> org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException
> -- Time elapsed: 127.8 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.container.OzoneTestHelper.waitForContainerClose(OzoneTestHelper.java:358)
> at
> org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException(TestFailureHandlingByClient.java:412)
> {code}
> h3. Root cause
> The previous test in the class shut down two datanodes of a pipeline, and
> {{restartDownDataNodes}} restarted them before this test.
> {{waitForClusterToBeReady}} only waits for the datanodes to register with
> SCM; it does not wait for their Ratis groups to elect a leader. In the
> failing run the surviving peer of that pipeline kept getting {{UNAVAILABLE:
> io exception}} on vote requests to the restarted peers, so the group had no
> leader from 03:41:58 until 03:42:36.
> SCM still considered the pipeline open, so the first block of the test key
> (container 4) was allocated on it. The write timed out with
> {{RaftRetryFailureException}} after about 27s and the client moved the data
> to a new block on another pipeline. The test then read the container ID from
> {{streamEntryList.get(0)}}, which is still container 4. That container was
> never created on any datanode, so {{waitForContainerClose}} timed out waiting
> for it to be present (100s).
> The same leaderless pipeline can also break the other tests in the class,
> which assume their first block lands on a healthy pipeline.
> h3. Fix
> In {{restartDownDataNodes}}, after the cluster is ready, wait until every
> Ratis group on each restarted datanode has a leader before the next test
> starts writing.
> - https://github.com/apache/ozone/actions/runs/38341370639/job/115115959763
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
