Chu Cheng Li created HDDS-16628:
-----------------------------------
Summary: SCMContainerPlacementRackAware gives up before trying the
other used node's rack
Key: HDDS-16628
URL: https://issues.apache.org/jira/browse/HDDS-16628
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
When ReplicationManager adds a replica for a RATIS container whose used
replicas are on two racks, {{SCMContainerPlacementRackAware.chooseNode}} looks
for a node on the rack of each used node (affinity node) in turn. It picks
random nodes and gives up after {{MAX_RETRY}} (3) of them fail {{isValidNode}}.
Nodes that are not writable, such as DECOMMISSIONING, DECOMMISSIONED or
IN_MAINTENANCE nodes, stay in the network topology until they go dead, and each
one picked uses up a retry.
If the first used node's rack has 3 or more such nodes and no valid node, every
attempt runs out of retries on that rack and throws "No satisfied datanode to
meet the space constrains". The other used node's rack and the fallback are
never tried. The container stays under-replicated, and decommissioning a node
that holds one of its replicas never completes. The same happens when the nodes
on that rack are out of space.
An SCM simulation found this: three decommissioned but still running nodes on
one rack left 15-26 containers under-replicated and a node stuck in
DECOMMISSIONING.
Fix: give each affinity node's rack its own retries. When they run out, move on
to the next affinity node's rack, and then to the fallback. Also reword the
error, which blamed space even when the nodes were out of service.
Tests: two new tests in {{TestSCMContainerPlacementRackAware}}, which fail
without the fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
