[
https://issues.apache.org/jira/browse/HDDS-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16628:
-----------------------------------
Assignee: Chu Cheng Li
> SCMContainerPlacementRackAware gives up before trying the other used node's
> rack
> --------------------------------------------------------------------------------
>
> Key: HDDS-16628
> URL: https://issues.apache.org/jira/browse/HDDS-16628
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> When ReplicationManager adds a replica for a RATIS container whose used
> replicas are on two racks, {{SCMContainerPlacementRackAware.chooseNode}}
> looks for a node on the rack of each used node (affinity node) in turn. It
> picks random nodes and gives up after {{MAX_RETRY}} (3) of them fail
> {{isValidNode}}. Nodes that are not writable, such as DECOMMISSIONING,
> DECOMMISSIONED or IN_MAINTENANCE nodes, stay in the network topology until
> they go dead, and each one picked uses up a retry.
> If the first used node's rack has 3 or more such nodes and no valid node,
> every attempt runs out of retries on that rack and throws "No satisfied
> datanode to meet the space constrains". The other used node's rack and the
> fallback are never tried. The container stays under-replicated, and
> decommissioning a node that holds one of its replicas never completes. The
> same happens when the nodes on that rack are out of space.
> An SCM simulation found this: three decommissioned but still running nodes on
> one rack left 15-26 containers under-replicated and a node stuck in
> DECOMMISSIONING.
> Fix: give each affinity node's rack its own retries. When they run out, move
> on to the next affinity node's rack, and then to the fallback. Also reword
> the error, which blamed space even when the nodes were out of service.
> Tests: two new tests in {{TestSCMContainerPlacementRackAware}}, which fail
> without the fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
