[
https://issues.apache.org/jira/browse/HDDS-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-8336:
------------------------------------
Fix Version/s: 1.4.0
> ReplicationManager: RatisUnderReplicationHandler should partially recover the
> container if not enough nodes
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-8336
> URL: https://issues.apache.org/jira/browse/HDDS-8336
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.4.0
>
>
> In RatisUnderReplicationHandler, if the container is under-replicated by more
> than 1 replica, then we need to select 2 or more new copies. In the code
> getTargets():
> {code}
> private List<DatanodeDetails> getTargets(
> RatisContainerReplicaCount replicaCount,
> List<ContainerReplicaOp> pendingOps) throws IOException {
> // DNs that already have replicas cannot be targets and should be excluded
> final List<DatanodeDetails> excludeList =
> replicaCount.getReplicas().stream()
> .map(ContainerReplica::getDatanodeDetails)
> .collect(Collectors.toList());
> // DNs that are already waiting to receive replicas cannot be targets
> final List<DatanodeDetails> pendingReplication =
> pendingOps.stream()
> .filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
> ContainerReplicaOp.PendingOpType.ADD)
> .map(ContainerReplicaOp::getTarget)
> .collect(Collectors.toList());
> excludeList.addAll(pendingReplication);
> /*
> Ensure that target datanodes have enough space to hold a complete
> container.
> */
> final long dataSizeRequired =
> Math.max(replicaCount.getContainer().getUsedBytes(),
> currentContainerSize);
> return placementPolicy.chooseDatanodes(excludeList, null,
> replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
> }
> {code}
> We ask the placement policy for the required number of nodes. If it cannot
> provide the required number (eg we want 2, but it can only find 1 spare
> node), it will throw a SCMException. In that case, we should try again
> requesting one less node to see if we can partially recover. If that fails,
> try again with another less node until we reach zero.
> If we successfully send a command for some but not all new copies, then we
> should throw an exception at the end so the container is re-queued on the
> under-rep queue to be tried again in a short while.
> Note the class `MisReplicationHandler.getTargetDatanodes()` currently has
> logic like this, but it does not throw at the end to trigger a re-queue -
> HDDS-8337
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
