Chu Cheng Li created HDDS-16633:
-----------------------------------
Summary: Datanode can go missing from the network topology if it
re-registers while SCM is removing it as DEAD
Key: HDDS-16633
URL: https://issues.apache.org/jira/browse/HDDS-16633
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
When a datanode registers again with a new IP address, hostname, ports or
version, {{SCMNodeManager#register}} replaces its object in the network
topology with {{clusterMap.update}}, then replaces its {{DatanodeInfo}} with
{{NodeStateManager#updateNode}}. The IP address and hostname case has always
done this, and HDDS-16630 adds it for ports and version. This is not
synchronized with the code that removes a DEAD datanode from the topology:
* On master, {{DeadNodeHandler}} checks {{NetworkTopology#contains}}, then
calls {{remove}}.
* With HDDS-16623, {{NodeStateManager#checkNodesHealth}} marks the node DEAD,
then removes it from the topology.
If the re-registration runs between those two steps, the removal takes out the
object that the re-registration just added, because {{InnerNodeImpl#remove}}
removes whatever child has the node's name. The datanode ends up HEALTHY but
missing from the topology, so placement never chooses it. It is only added back
if it goes DEAD and recovers, or when SCM restarts.
{{NetworkTopology#contains}} still returns true for its {{DatanodeInfo}},
because its parent pointer still points at the rack. On master,
{{DeadNodeHandler}} also throws {{IllegalStateException}} from its
{{Preconditions.checkState(node.getParent() == null)}} check.
The window is small: the datanode has to register again at the moment SCM
declares it DEAD. Both variants were reproduced in unit tests that run the
re-registration from inside {{NetworkTopology#remove}}. Afterwards the node was
HEALTHY, {{getNumOfLeafNode("")}} was 0, and {{contains()}} still returned true.
Found while working on HDDS-16630.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
