[
https://issues.apache.org/jira/browse/HDDS-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16630:
-----------------------------------
Assignee: Chu Cheng Li
> SCM does not update the network topology when a datanode re-registers with
> new ports or a new version
> -----------------------------------------------------------------------------------------------------
>
> Key: HDDS-16630
> URL: https://issues.apache.org/jira/browse/HDDS-16630
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
>
> When a datanode that is already registered registers again,
> {{SCMNodeManager#register}} handles three kinds of change:
> # IP address or hostname changed: it calls {{clusterMap.update(oldNode,
> datanodeDetails)}} and then {{nodeStateManager.updateNode(...)}}.
> # Version changed: it only calls {{nodeStateManager.updateNode(...)}}.
> # Ports changed: it only calls {{nodeStateManager.updateNode(...)}}.
> {{NodeStateManager#updateNode}} stores a new {{DatanodeInfo}} for the node.
> In cases 2 and 3 nothing updates the network topology, which causes three
> problems:
> * The topology still holds the old node object, with the old ports and
> version. Placement policies such as {{SCMContainerPlacementRackAware}} and
> {{SCMContainerPlacementRackScatter}} return the node object they get from
> {{NetworkTopology#chooseRandom}}, so new pipelines and replication targets
> can still get the old ports.
> * The new {{DatanodeInfo}} returned by {{getNode()}} and {{getAllNodes()}}
> has no parent in the topology, so {{NetworkTopology#contains}} returns false
> for it. When the node later goes DEAD, {{DeadNodeHandler}} does not find it,
> and the old object stays in the topology.
> * A DEAD node has already been removed from the topology. If it registers
> again with new ports or a new version, it becomes HEALTHY but is never added
> back.
> To reproduce: register a datanode, register it again with a different Ratis
> port, and check {{clusterMap.contains(nodeManager.getNode(id))}}. It returns
> false.
> The SCM deterministic simulation test (HDDS-16627) found this. With its fault
> that restarts a datanode on new ports turned on, every seed fails with "dnX
> is HEALTHY but missing from the network topology".
> Fix: call {{clusterMap.update(oldNode, datanodeDetails)}} in the version and
> port branches too, as the IP/hostname branch already does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
