devmadhuu commented on code in PR #11866:
URL: https://github.com/apache/ozone/pull/11866#discussion_r4535305109


##########
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerClusterAnalyzer.java:
##########
@@ -40,6 +43,102 @@ public final class TestContainerBalancerClusterAnalyzer {
 
   private static final double THRESHOLD = 0.10;
 
+  @Test
+  void testOppositeStorageTypeImbalancesAreNotHiddenByWholeNodeTotals() {

Review Comment:
   Added the test.  The implementation combines source datanode IDs using a 
set, so A is counted once even when it is a source for both SSD and DISK.



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerClusterAnalyzer.java:
##########
@@ -195,12 +237,63 @@ public static ContainerBalancerClusterSnapshot analyze(
         .setTargetCount(targets.size())
         .setTotalOverUtilizedBytes(totalOverUtilizedBytes)
         .setTotalUnderUtilizedBytes(totalUnderUtilizedBytes)
+        .setBytesToMove(storageType != null && (sources.isEmpty() || 
targets.isEmpty()) ? 0 : totalOverUtilizedBytes)
+        .setSourceDatanodes(sourceDatanodes)
+        .setTargetDatanodes(targetDatanodes)
         .setImbalance(imbalance)
         .setTopSourceNodes(topNodes(sources, topNodeLimit))
         .setBottomTargetNodes(topNodes(targets, topNodeLimit))
         .build();
   }
 
+  private static ContainerBalancerClusterSnapshot 
combineStorageTypes(ContainerBalancerClusterSnapshot overall,
+      Map<StorageType, ContainerBalancerClusterSnapshot> snapshots, int 
topNodeLimit, boolean incomplete) {
+    Set<String> sources = new HashSet<>();
+    Set<String> targets = new HashSet<>();
+    Map<String, NodeUtilization> topSources = new HashMap<>();
+    Map<String, NodeUtilization> bottomTargets = new HashMap<>();
+    long overBytes = 0, underBytes = 0, bytesToMove = 0;
+    double max = 0, min = 0, imbalance = -1;
+    for (ContainerBalancerClusterSnapshot snapshot : snapshots.values()) {
+      sources.addAll(snapshot.getSourceDatanodes());
+      targets.addAll(snapshot.getTargetDatanodes());
+      for (NodeUtilization node : snapshot.getTopSourceNodes()) {
+        topSources.merge(node.getDatanodeId(), node,
+            (a, b) -> a.getUtilization() >= b.getUtilization() ? a : b);
+      }
+      for (NodeUtilization node : snapshot.getBottomTargetNodes()) {
+        bottomTargets.merge(node.getDatanodeId(), node,
+            (a, b) -> a.getUtilization() <= b.getUtilization() ? a : b);
+      }
+      overBytes += snapshot.getTotalOverUtilizedBytes();
+      underBytes += snapshot.getTotalUnderUtilizedBytes();
+      bytesToMove += snapshot.getBytesToMove();
+      if (snapshot.getImbalance() > imbalance) {

Review Comment:
   yes, this has been handled as part of 1st review comment.



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerAdvisor.java:
##########
@@ -154,6 +154,15 @@ private static ContainerBalancerEstimation 
estimateForProfile(OzoneConfiguration
           maxSizeLeavingSource,
           maxSizeEnteringTarget,
           involved);
+      if (!snapshot.getStorageTypeSnapshots().isEmpty()) {
+        long storageTypeCapacity = 0;
+        for (ContainerBalancerClusterSnapshot type : 
snapshot.getStorageTypeSnapshots().values()) {

Review Comment:
   Done.



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerAdvisor.java:
##########
@@ -154,6 +154,15 @@ private static ContainerBalancerEstimation 
estimateForProfile(OzoneConfiguration
           maxSizeLeavingSource,
           maxSizeEnteringTarget,
           involved);
+      if (!snapshot.getStorageTypeSnapshots().isEmpty()) {
+        long storageTypeCapacity = 0;

Review Comment:
   Agreed. This variable sums the estimated bytes movable in one iteration 
across the storage types



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to