[
https://issues.apache.org/jira/browse/HDDS-17363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-17363:
--------------------------------
Description:
*Problem*
The EC mis-replication check and {{MisReplicationHandler}} count all replicas
when they count racks. This includes the replicas on decommissioning and
decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on
datanodes that are in service and healthy, when it selects the copy to delete.
Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12.
dn12 is DECOMMISSIONED and still up.
||Rack||Replicas||
|rack-0|none|
|rack-1|dn2 (index 3), dn6 (index 4)|
|rack-2|dn3 (index 1), dn7 (index 5)|
|rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
# The container uses 3 of the 4 racks. Thus, it is mis-replicated.
# {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from
dn8 to a datanode on rack-0.
# Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12.
Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new
copy on rack-0.
# The container is back at step 1.
The choices in step 2 and step 3 depend on the hash order of the rack names.
With the names /rack-0 to /rack-3 the loop occurs.
*Effect*
SCM copies a replica and deletes it again, over and over, while the
decommissioned datanode stays up. The container stays MIS_REPLICATED. In the
SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15
minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
*Fix*
Options:
* Do not count replicas on decommissioning and decommissioned datanodes for
placement in the mis-replication check and handler. These replicas go away, and
the over-replication handler does not count them either. A trial change in
{{MisReplicationHandler}} only passes the new test, the existing replication
handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027.
With it and a trial fix for HDDS-17364, the simulation passes 100 of 100 seeds
with 1 SCM and 100 of 100 with 3 SCMs.
* Or let {{ECOverReplicationHandler}} count all replicas per rack when it
selects the copy to delete.
*Tests*
{{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn
with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
{noformat}
Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0,
delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0,
delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0,
delete index 2 on dn9 on /rack-0]
{noformat}
was:
*Problem*
The EC mis-replication check and {{MisReplicationHandler}} count all replicas
when they count racks. This includes the replicas on decommissioning and
decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas on
datanodes that are in service and healthy, when it selects the copy to delete.
Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on dn12.
dn12 is DECOMMISSIONED and still up.
||Rack||Replicas||
|rack-0|none|
|rack-1|dn2 (index 3), dn6 (index 4)|
|rack-2|dn3 (index 1), dn7 (index 5)|
|rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
# The container uses 3 of the 4 racks. Thus, it is mis-replicated.
# {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2 from
dn8 to a datanode on rack-0.
# Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12.
Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new
copy on rack-0.
# The container is back at step 1.
The choices in step 2 and step 3 depend on the hash order of the rack names.
With the names /rack-0 to /rack-3 the loop occurs.
*Effect*
SCM copies a replica and deletes it again, over and over, while the
decommissioned datanode stays up. The container stays MIS_REPLICATED. In the
SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in 15
minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
*Fix*
Options:
* Do not count replicas on decommissioning and decommissioned datanodes for
placement in the mis-replication check and handler. These replicas go away, and
the over-replication handler does not count them either. A trial change in
{{MisReplicationHandler}} only passes the new test, the existing replication
handler and ReplicationManager tests (171), and simulation seeds 2006 and 2027.
With it and a trial fix for the safe mode bug that the simulation also found,
the simulation passes 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
* Or let {{ECOverReplicationHandler}} count all replicas per rack when it
selects the copy to delete.
*Tests*
{{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in turn
with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
{noformat}
Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0,
delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on /rack-0,
delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9 on /rack-0,
delete index 2 on dn9 on /rack-0]
{noformat}
> EC mis-replication and over-replication handlers undo each other when a
> decommissioned datanode keeps a replica
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17363
> URL: https://issues.apache.org/jira/browse/HDDS-17363
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Priority: Major
>
> *Problem*
> The EC mis-replication check and {{MisReplicationHandler}} count all replicas
> when they count racks. This includes the replicas on decommissioning and
> decommissioned datanodes. {{ECOverReplicationHandler}} counts only replicas
> on datanodes that are in service and healthy, when it selects the copy to
> delete.
> Example with 4 racks and rs-3-2. Index 3 has a copy on dn2 and a copy on
> dn12. dn12 is DECOMMISSIONED and still up.
> ||Rack||Replicas||
> |rack-0|none|
> |rack-1|dn2 (index 3), dn6 (index 4)|
> |rack-2|dn3 (index 1), dn7 (index 5)|
> |rack-3|dn8 (index 2), dn12 (index 3, DECOMMISSIONED)|
> # The container uses 3 of the 4 racks. Thus, it is mis-replicated.
> # {{MisReplicationHandler}} counts 2 replicas on rack-3. It copies index 2
> from dn8 to a datanode on rack-0.
> # Index 2 now has 2 copies. {{ECOverReplicationHandler}} does not count dn12.
> Thus, it sees 1 replica on rack-0 and 1 replica on rack-3. It deletes the new
> copy on rack-0.
> # The container is back at step 1.
> The choices in step 2 and step 3 depend on the hash order of the rack names.
> With the names /rack-0 to /rack-3 the loop occurs.
> *Effect*
> SCM copies a replica and deletes it again, over and over, while the
> decommissioned datanode stays up. The container stays MIS_REPLICATED. In the
> SCM simulation (HDDS-16627), seed 2006 sent 15 commands for one container in
> 15 minutes. Seed 2027 (3 SCMs) did the same for 10 containers.
> *Fix*
> Options:
> * Do not count replicas on decommissioning and decommissioned datanodes for
> placement in the mis-replication check and handler. These replicas go away,
> and the over-replication handler does not count them either. A trial change
> in {{MisReplicationHandler}} only passes the new test, the existing
> replication handler and ReplicationManager tests (171), and simulation seeds
> 2006 and 2027. With it and a trial fix for HDDS-17364, the simulation passes
> 100 of 100 seeds with 1 SCM and 100 of 100 with 3 SCMs.
> * Or let {{ECOverReplicationHandler}} count all replicas per rack when it
> selects the copy to delete.
> *Tests*
> {{TestECMisReplicationWithDecommissionedReplica}} runs the two handlers in
> turn with a real {{SCMContainerPlacementRackScatter}}. It fails on master:
> {noformat}
> Still mis-replicated after [replicate index 2 from dn8 to dn1 on /rack-0,
> delete index 2 on dn1 on /rack-0, replicate index 2 from dn8 to dn5 on
> /rack-0, delete index 2 on dn5 on /rack-0, replicate index 2 from dn8 to dn9
> on /rack-0, delete index 2 on dn9 on /rack-0]
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
