Chu Cheng Li created HDDS-16632:
-----------------------------------
Summary: SCM loses a datanode's container list when the datanode
re-registers with a new address, ports or version
Key: HDDS-16632
URL: https://issues.apache.org/jira/browse/HDDS-16632
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
When a registered datanode registers again with a new IP address, hostname,
ports or version, {{SCMNodeManager#register}} calls
{{NodeStateManager#updateNode}}. {{NodeStateMap#updateNode}} then replaces the
node's {{DatanodeEntry}} with a new one, which starts with an empty container
list. A rolling upgrade changes the version of every datanode, so each restart
during the upgrade takes this path, and so does any restart that changes a
datanode's IP address.
SCM uses this list to find replicas that disappeared from a datanode:
* {{ContainerReportHandler}} compares each full container report with
{{nodeManager.getContainers(dn)}}, and removes the replicas of containers that
are in the list but not in the report.
* {{DeadNodeHandler#removeContainerReplicas}} removes a dead datanode's
replicas using the same list.
So if a datanode loses a replica while it is down, for example on a failed
disk, and then registers again with one of these changes, SCM never removes
that replica. The report sent with the registration no longer includes it, and
neither does the emptied list. SCM keeps counting the replica, so
{{ReplicationManager}} does not treat the container as under-replicated. The
replica is not removed when the datanode goes DEAD either.
{{DatanodeAdminMonitorImpl#checkContainersReplicatedOnNode}} reads the same
list. Until the container report sent with the registration has been processed,
a datanode that is decommissioning or entering maintenance looks like it has no
containers to wait for. This part was found by reading the code and has not
been reproduced.
A unit test with the real {{SCMNodeManager}} and {{ContainerReportHandler}}
reproduces it:
# Register a datanode and process a full container report from it with a CLOSED
container that also has replicas on two other datanodes.
# Register the datanode again with a different Ratis port.
# Process full container reports from it that no longer include the container.
SCM still lists 3 replicas, also after the datanode goes DEAD and
{{DeadNodeHandler}} runs. Without the port change, the same reports bring it
down to 2.
Found while working on HDDS-16630.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
