Chu Cheng Li created HDDS-16630:
-----------------------------------
Summary: SCM does not update the network topology when a datanode
re-registers with new ports or a new version
Key: HDDS-16630
URL: https://issues.apache.org/jira/browse/HDDS-16630
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
When a datanode that is already registered registers again,
{{SCMNodeManager#register}} handles three kinds of change:
# IP address or hostname changed: it calls {{clusterMap.update(oldNode,
datanodeDetails)}} and then {{nodeStateManager.updateNode(...)}}.
# Version changed: it only calls {{nodeStateManager.updateNode(...)}}.
# Ports changed: it only calls {{nodeStateManager.updateNode(...)}}.
{{NodeStateManager#updateNode}} stores a new {{DatanodeInfo}} for the node. In
cases 2 and 3 nothing updates the network topology, which causes three problems:
* The topology still holds the old node object, with the old ports and version.
Placement policies such as {{SCMContainerPlacementRackAware}} and
{{SCMContainerPlacementRackScatter}} return the node object they get from
{{NetworkTopology#chooseRandom}}, so new pipelines and replication targets can
still get the old ports.
* The new {{DatanodeInfo}} returned by {{getNode()}} and {{getAllNodes()}} has
no parent in the topology, so {{NetworkTopology#contains}} returns false for
it. When the node later goes DEAD, {{DeadNodeHandler}} does not find it, and
the old object stays in the topology.
* A DEAD node has already been removed from the topology. If it registers again
with new ports or a new version, it becomes HEALTHY but is never added back.
To reproduce: register a datanode, register it again with a different Ratis
port, and check {{clusterMap.contains(nodeManager.getNode(id))}}. It returns
false.
The SCM deterministic simulation test (HDDS-16627) found this. With its fault
that restarts a datanode on new ports turned on, every seed fails with "dnX is
HEALTHY but missing from the network topology".
Fix: call {{clusterMap.update(oldNode, datanodeDetails)}} in the version and
port branches too, as the IP/hostname branch already does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
