Chu Cheng Li created HDDS-16629:
-----------------------------------
Summary: Decommissioning a whole rack in a 2-rack cluster leaves
containers under-replicated
Key: HDDS-16629
URL: https://issues.apache.org/jira/browse/HDDS-16629
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
In a cluster with two racks, when every node of one rack is being
decommissioned, {{SCMContainerPlacementRackAware}} looks for the new replica on
a rack other than those of the used replicas. Only out-of-service nodes are
left there. They use up the retries, and the placement throws before its
fallback, which allows the used replicas' rack, is reached. The containers stay
under-replicated and the decommission never completes.
Falling back when the retries run out is not enough on its own.
{{validateContainerPlacement}} still counts the decommissioned rack, so a
container whose replicas all end up on the remaining rack is mis-replicated.
{{MisReplicationHandler}} then gets a same-rack target from the fallback and
copies to it. The container becomes over-replicated, a replica is deleted, and
it is mis-replicated again, in a loop.
Fix:
* {{SCMContainerPlacementRackAware.chooseNode}}: when the retries run out at
rack level and fallback is enabled, stop requiring a different rack, with new
retries.
* {{MisReplicationHandler}}: only copy when the targets reduce the
mis-replication or satisfy the placement policy; otherwise log and skip.
Tests: new unit tests for both parts, which fail without the fix. In an SCM
simulation of 100 random seeds over 2, 3, 4 and 12 racks, master had 6 stuck
decommissions. With this change and HDDS-16628 there were none, and no
replicate or delete commands were sent after the cluster settled.
Depends on HDDS-16628.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
