Siyao Meng created HDDS-17190:
---------------------------------
Summary: Intermittent failure in
TestDecommissionAndMaintenance#testSingleNodeWithOpenPipelineCanGotoMaintenance
Key: HDDS-17190
URL: https://issues.apache.org/jira/browse/HDDS-17190
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 320.6 s
<<< FAILURE! -- in
org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance
[ERROR]
org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance
-- Time elapsed: 59.78 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.hdds.scm.node.NodeTestUtil.waitForDnToReachPersistedOpState(NodeTestUtil.java:98)
at
org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance(TestDecommissionAndMaintenance.java:487)
{code}
h3. Root cause
{{waitForDnToReachPersistedOpState}} polls {{getPersistedOpState()}} on the
{{DatanodeDetails}} object it is given. The test passes {{dn}}, captured from
{{nm.getNode(dnID)}} before the datanode restarts. When a restarted datanode
re-registers with a different port set, {{SCMNodeManager#register}} replaces
the stored {{DatanodeInfo}} (the {{portsChanged}} path added by HDDS-12991). In
the failing run the HTTP port was missing from the initial registration and
present on re-registration:
{noformat}
SCMNodeManager - Updating ports for registered datanode ca3c16ee-...:
[STANDALONE=15350, ..., CLIENT_RPC=15349] -> [STANDALONE=15350, ...,
HTTP=15348, CLIENT_RPC=15349]
{noformat}
After that SCM updates the persisted state on the new object (the log shows
{{IN_MAINTENANCE -> IN_SERVICE}}), while the test keeps polling the stale
{{dn}} until it times out. The neighbouring assertions already use {{newDn}}.
{{testNodeWithOpenPipelineCanBeDecommissionedAndRecommissioned}} has the same
stale reference ({{toDecommission}}) after a restart.
h3. Fix
Look up the node from {{NodeManager}} after the restart ({{newDn}} /
{{nm.getNode(dnID)}}) before waiting for the persisted state.
- https://github.com/apache/ozone/actions/runs/39445520577/job/118801747190
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
