Chu Cheng Li created HDDS-17363:
-----------------------------------
Summary: EC mis-replication and over-replication handlers undo
each other when a decommissioned datanode keeps a replica
Key: HDDS-17363
URL: https://issues.apache.org/jira/browse/HDDS-17363
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
*Problem*
The EC mis-replication check and {{MisReplicationHandler}} count all replicas
when they count racks. This includes the replicas on decommissioning and
decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on
datanodes that are in service and healthy, when it selects the copy to delete.
Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12.
dn12 is DECOMMISSIONED and still up.
||Rack||Replicas||
|rack-0|none|
|rack-1|dn2 (index 3), dn6 (index 4)|
|rack-2|dn3 (index 1), dn7 (index 5)|
|rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
# The container uses 3 of the 4 racks. Thus, it is mis-replicated.
# {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from
dn8 to a datanode on rack-0.
# Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12.
Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new
copy on rack-0.
# The container is back at step 1.
The choices in step 2 and step 3 depend on the hash order of the rack names.
With the names /rack-0 to /rack-3 the loop occurs.
*Effect*
SCM copies a replica and deletes it again, over and over, while the
decommissioned datanode stays up. The container stays MIS_REPLICATED. In the
SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15
minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
*Fix*
Options:
* Do not count replicas on decommissioning and decommissioned datanodes for
placement in the mis-replication check and handler. These replicas go away, and
the over-replication handler does not count them either. A trial change in
{{MisReplicationHandler}} only passes the new test, the existing replication
handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027.
With it and a trial fix for the safe mode bug that the simulation also found,
the simulation passes 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
* Or let {{ECOverReplicationHandler}} count all replicas per rack when it
selects the copy to delete.
*Tests*
{{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn
with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
{noformat}
Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0,
delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0,
delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0,
delete index 2 on dn9 on /rack-0]
{noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
