Stephen O'Donnell created HDDS-8336:
---------------------------------------
Summary: ReplicationManager: RatisUnderReplicationHandler should
partially recover the container if not enough nodes
Key: HDDS-8336
URL: https://issues.apache.org/jira/browse/HDDS-8336
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM
Reporter: Stephen O'Donnell
In RatisUnderReplicationHandler, if the container is under-replicated by more
than 1 replica, then we need to select 2 or more new copies. In the code
getTargets():
{code}
private List<DatanodeDetails> getTargets(
RatisContainerReplicaCount replicaCount,
List<ContainerReplicaOp> pendingOps) throws IOException {
// DNs that already have replicas cannot be targets and should be excluded
final List<DatanodeDetails> excludeList =
replicaCount.getReplicas().stream()
.map(ContainerReplica::getDatanodeDetails)
.collect(Collectors.toList());
// DNs that are already waiting to receive replicas cannot be targets
final List<DatanodeDetails> pendingReplication =
pendingOps.stream()
.filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
ContainerReplicaOp.PendingOpType.ADD)
.map(ContainerReplicaOp::getTarget)
.collect(Collectors.toList());
excludeList.addAll(pendingReplication);
/*
Ensure that target datanodes have enough space to hold a complete
container.
*/
final long dataSizeRequired =
Math.max(replicaCount.getContainer().getUsedBytes(),
currentContainerSize);
return placementPolicy.chooseDatanodes(excludeList, null,
replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
}
{code}
We ask the placement policy for the required number of nodes. If it cannot
provide the required number (eg we want 2, but it can only find 1 spare node),
it will throw a SCMException. In that case, we should try again requesting one
less node to see if we can partially recover. If that fails, try again with
another less node until we reach zero.
If we successfully send a command for some but not all new copies, then we
should throw an exception at the end so the container is re-queued on the
under-rep queue to be tried again in a short while.
Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic
like this, but it does not throw at the end to trigger a re-queue - JIRA_TO_FIX
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
