[
https://issues.apache.org/jira/browse/HDDS-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16629:
-----------------------------------
Assignee: Chu Cheng Li
> Decommissioning a whole rack in a 2-rack cluster leaves containers
> under-replicated
> -----------------------------------------------------------------------------------
>
> Key: HDDS-16629
> URL: https://issues.apache.org/jira/browse/HDDS-16629
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> In a cluster with two racks, when every node of one rack is being
> decommissioned, {{SCMContainerPlacementRackAware}} looks for the new replica
> on a rack other than those of the used replicas. Only out-of-service nodes
> are left there. They use up the retries, and the placement throws before its
> fallback, which allows the used replicas' rack, is reached. The containers
> stay under-replicated and the decommission never completes.
> Falling back when the retries run out is not enough on its own.
> {{validateContainerPlacement}} still counts the decommissioned rack, so a
> container whose replicas all end up on the remaining rack is mis-replicated.
> {{MisReplicationHandler}} then gets a same-rack target from the fallback and
> copies to it. The container becomes over-replicated, a replica is deleted,
> and it is mis-replicated again, in a loop.
> Fix:
> * {{SCMContainerPlacementRackAware.chooseNode}}: when the retries run out at
> rack level and fallback is enabled, stop requiring a different rack, with new
> retries.
> * {{MisReplicationHandler}}: only copy when the targets reduce the
> mis-replication or satisfy the placement policy; otherwise log and skip.
> Tests: new unit tests for both parts, which fail without the fix. In an SCM
> simulation of 100 random seeds over 2, 3, 4 and 12 racks, master had 6 stuck
> decommissions. With this change and HDDS-16628 there were none, and no
> replicate or delete commands were sent after the cluster settled.
> Depends on HDDS-16628.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
