sodonnel commented on code in PR #3836:
URL: https://github.com/apache/ozone/pull/3836#discussion_r1001687051
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ECUnderReplicationHandler.java:
##########
@@ -353,8 +386,77 @@ private void processMaintenanceOnlyIndexes(
DatanodeDetails target = iterator.next();
commands.put(target, replicateCommand);
additionalMaintenanceCopiesNeeded -= 1;
+ maintIndexes.remove(replica.getReplicaIndex());
+ createdIndexes.add(replica.getReplicaIndex());
+ }
+ }
+ }
+
+ /**
+ * Function processes Replicas to fix placement policy issues.
+ * @param replicas
+ * @param deletionInFlight
+ * @param container
+ * @param excludedNodes
+ * @param sources
+ * @param createdIndexes
+ * @param placementStatus
+ * @throws IOException
+ */
+ private Map<DatanodeDetails, SCMCommand<?>> processMisreplication(
+ Set<ContainerReplica> replicas,
+ List<DatanodeDetails> deletionInFlight, ContainerInfo container,
+ List<DatanodeDetails> excludedNodes,
+ Map<Integer, Pair<ContainerReplica, NodeStatus>> sources,
+ Set<Integer> createdIndexes, ContainerPlacementStatus
placementStatus)
+ throws IOException {
+ if (placementStatus.isPolicySatisfied() ||
+ placementStatus.misReplicationCount() <= createdIndexes.size()) {
+ return Collections.emptyMap();
+ }
+ Map<DatanodeDetails, SCMCommand<?>> commands = new HashMap<>();
+ List<Integer> sortedReplicaIdx =
+ getSourcesStream(replicas, deletionInFlight)
+ .collect(Collectors.groupingBy(
+ r -> r.getLeft().getReplicaIndex(),
+ Collectors.counting()))
+ .entrySet().stream()
+ .sorted(Comparator.comparingLong(Map.Entry::getValue))
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ int additionalCopiesForPlacementStatus =
+ placementStatus.misReplicationCount() - createdIndexes.size();
+ List<DatanodeDetails> targets = getTargetDatanodes(excludedNodes,
container,
Review Comment:
Is it possible that this call will give some target that do not improvement
the placement? Eg for some reason we require 5 racks, but the container is on
4. Then this call returns a new node, but it duplicates one of the racks we are
already on, so the placement is still bad. The rack scatter is (I think) best
effort at picking racks - it will give a node that does not meet the policy if
it cannot find anything better.
So we probably need a check to see if the target improves the situation
before scheduling the commands to do the copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
