Chu Cheng Li created HDDS-17362:
-----------------------------------
Summary: RackScatter counts a rack of out-of-service datanodes as
available
Key: HDDS-17362
URL: https://issues.apache.org/jira/browse/HDDS-17362
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
*Problem*
{{SCMContainerPlacementRackScatter}} spreads the replicas of an EC container
over as many racks as it can. It gets the number of racks from the network
topology. Decommissioning, decommissioned and maintenance datanodes stay in the
topology. Thus, a rack whose datanodes are all out of service still counts as a
rack that can take a replica.
HDDS-9125 removes a rack from the count only when all its datanodes are in the
excluded list. A datanode that is out of service but holds no replica of the
container is not in the excluded list. Thus, its rack still counts.
*Effect*
# SCM cannot create an EC pipeline while one rack is out of service. Example: 3
racks with 3 datanodes each, and all datanodes on the third rack are in
decommission. For rs-3-2 the policy allows 2 replicas per rack (5 replicas on 3
racks). The 2 racks in service can take only 4 replicas. The placement fails
with "Chosen nodes size: 4, but required nodes to choose: 5 do not match".
# A decommission of a whole rack does not finish for EC containers. SCM must
copy each replica on that rack to another rack. When every other rack already
has a replica, the policy still asks for a rack without one, because it counts
the rack in decommission. No datanode there can take the copy. Thus, the
placement fails with "Chosen nodes size from Unique Racks: 0, but required
nodes to choose from Unique Racks: 1 do not match", and the datanodes stay
DECOMMISSIONING.
*Fix*
Count a rack only if at least one of its datanodes can take a new replica (in
service and healthy). Use the same rule in the final check that the new
placement does not make mis-replication worse.
*Tests*
Two new tests in {{TestSCMContainerPlacementRackScatter}} fail on master:
{{testNewPlacementSkipsRackWithOnlyOutOfServiceNodes}} and
{{testRackWithOnlyOutOfServiceNodesReducesRackCount}}. A candidate fix passes
them.
The SCM simulation (HDDS-16627) found this with EC containers and whole-rack
decommission. In its first EC batch, the candidate fix cut the failing seeds
from 30 to 20 of 100, and the "still DECOMMISSIONING" failures from 85 to 31.
The other failures had other causes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
