[
https://issues.apache.org/jira/browse/HDDS-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16934:
------------------------------
Status: Patch Available (was: Open)
> Intermittent failure in
> TestSCMNodeManager#testDatanodeFinalizedCounterExcludesNonHealthyNodes
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-16934
> URL: https://issues.apache.org/jira/browse/HDDS-16934
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR]
> org.apache.hadoop.hdds.scm.node.TestSCMNodeManager.testDatanodeFinalizedCounterExcludesNonHealthyNodes(NodeStatus)[2]
> <<< FAILURE!
> org.opentest4j.AssertionFailedError: IN_SERVICE(no expiry)-DEAD node should
> be excluded from total count ==> expected: <1> but was: <0>
> {code}
> Observed locally, about once in seven runs, only for the DEAD parameter.
> h3. Root cause
> The test (added in HDDS-15034) sets the stale node interval to 3s and the
> dead node interval to 6s, and drives the node state with fixed
> {{Thread.sleep}} calls. In the DEAD case the last heartbeat of the baseline
> node is followed by {{Thread.sleep(3000)}}, so that heartbeat is at least 3s
> old when the assertions run. {{NodeStateManager#checkNodesHealth}} only
> treats a node as healthy when its last heartbeat is within the stale node
> interval. If the health check (100ms period) fires between the end of the
> sleep and the assertion, the baseline node itself is marked STALE and
> {{getTotalHealthyDatanodes()}} returns 0 instead of 1.
> Adding a 150ms sleep before the assertions makes the DEAD case fail every
> time, with the baseline node reported as STALE.
> h3. Fix
> Replace the fixed sleeps with {{GenericTestUtils.waitFor}} that keeps
> heartbeating the baseline node until the other node reaches the expected
> state, as other tests in the same class already do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
