virajjasani commented on a change in pull request #3675:
URL: https://github.com/apache/hadoop/pull/3675#discussion_r773700080
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminBackoffMonitor.java
##########
@@ -189,6 +190,29 @@ public void run() {
* node will be removed from tracking by the pending cancel.
*/
processCancelledNodes();
+
+ // Having more nodes decommissioning than can be tracked will impact
decommissioning
+ // performance due to queueing delay
+ int numTrackedNodes = outOfServiceNodeBlocks.size();
+ int numQueuedNodes = getPendingNodes().size();
+ int numDecommissioningNodes = numTrackedNodes + numQueuedNodes;
+ if (numDecommissioningNodes > maxConcurrentTrackedNodes) {
+ LOG.warn(
+ "There are {} nodes decommissioning but only {} nodes will be
tracked at a time. "
Review comment:
nit: more readable `{} nodes are decommissioning but only.....`
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminMonitorBase.java
##########
@@ -151,4 +162,34 @@ public int getPendingNodeCount() {
public Queue<DatanodeDescriptor> getPendingNodes() {
return pendingNodes;
}
+
+ /**
+ * If node "is dead while in Decommission In Progress", it cannot be
decommissioned
+ * until it becomes healthy again. If there are more pendingNodes than can
be tracked
+ * & some unhealthy tracked nodes, then re-queue the unhealthy tracked nodes
+ * to avoid blocking decommissioning of healthy nodes.
+ *
+ * @param unhealthyDns The unhealthy datanodes which may be re-queued
+ * @param numDecommissioningNodes The total number of nodes being
decommissioned
+ * @return List of unhealthy nodes to be re-queued
+ */
+ Stream<DatanodeDescriptor> identifyUnhealthyNodesToRequeue(
Review comment:
nit: Since we are not only identifying but also returning the unhealthy
nodes to re-queue, shall we update this method name to
`identifyAndGetUnhealthyNodesToRequeue` or `getUnhealthyNodesToRequeue` ?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminMonitorBase.java
##########
@@ -35,12 +38,20 @@
public abstract class DatanodeAdminMonitorBase
implements DatanodeAdminMonitorInterface, Configurable {
+ /**
+ * Sort by lastUpdate time descending order, such that unhealthy
+ * nodes are de-prioritized given they cannot be decommissioned.
+ */
+ public static final Comparator<DatanodeDescriptor>
PENDING_NODES_QUEUE_COMPARATOR =
Review comment:
Let's remove public and keep default access modifier?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeAdminMonitorBase.java
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.PriorityQueue;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.hdfs.protocol.DatanodeID;
+
+public class TestDatanodeAdminMonitorBase {
+ public static final Logger LOG =
LoggerFactory.getLogger(TestDatanodeAdminMonitorBase.class);
+
+ // Sort by lastUpdate time descending order, such that unhealthy
+ // nodes are de-prioritized given they cannot be decommissioned.
+ private static final int NUM_DATANODE = 10;
+ private static final int[] UNORDERED_LAST_UPDATE_TIMES =
+ new int[] {0, 5, 2, 11, 0, 3, 1001, 5, 1, 103};
+ private static final int[] ORDERED_LAST_UPDATE_TIMES =
+ new int[] {1001, 103, 11, 5, 5, 3, 2, 1, 0, 0};
+ private static final int[] REVERSE_ORDER_LAST_UPDATE_TIMES =
+ new int[] {0, 0, 1, 2, 3, 5, 5, 11, 103, 1001};
+
+ private static final DatanodeDescriptor[] NODES;
+
+ static {
+ NODES = new DatanodeDescriptor[NUM_DATANODE];
+ for (int i = 0; i < NUM_DATANODE; i++) {
+ NODES[i] = new DatanodeDescriptor(DatanodeID.EMPTY_DATANODE_ID);
+ NODES[i].setLastUpdate(UNORDERED_LAST_UPDATE_TIMES[i]);
+ NODES[i].setLastUpdateMonotonic(UNORDERED_LAST_UPDATE_TIMES[i]);
+ }
+ }
+
+ /**
+ * Verify that DatanodeAdminManager pendingNodes priority queue
+ * correctly orders the nodes by lastUpdate time descending.
+ */
+ @Test
+ public void testPendingNodesQueueOrdering() {
+ final PriorityQueue<DatanodeDescriptor> pendingNodes =
+ new PriorityQueue<>(10,
+ DatanodeAdminMonitorBase.PENDING_NODES_QUEUE_COMPARATOR);
+
+ for (int i = 0; i < NUM_DATANODE; i++) {
+ pendingNodes.add(NODES[i]);
+ }
Review comment:
`pendingNodes.addAll(Arrays.asList(NODES).subList(0, NUM_DATANODE))`
would be better alternative using Collection copy API over manual copy
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
##########
@@ -270,12 +273,31 @@ private void check() {
// an invalid state.
LOG.warn("DatanodeAdminMonitor caught exception when processing node "
+ "{}.", dn, e);
- pendingNodes.add(dn);
+ getPendingNodes().add(dn);
toRemove.add(dn);
} finally {
iterkey = dn;
}
}
+
+ // Having more nodes decommissioning than can be tracked will impact
decommissioning
+ // performance due to queueing delay
+ int numTrackedNodes = outOfServiceNodeBlocks.size() - toRemove.size();
+ int numQueuedNodes = getPendingNodes().size();
+ int numDecommissioningNodes = numTrackedNodes + numQueuedNodes;
+ if (numDecommissioningNodes > maxConcurrentTrackedNodes) {
+ LOG.warn(
+ "There are {} nodes decommissioning but only {} nodes will be
tracked at a time. "
+ + "{} nodes are currently queued waiting to be decommissioned.",
+ numDecommissioningNodes, maxConcurrentTrackedNodes, numQueuedNodes);
Review comment:
Same as above, `{} nodes are decommissioning but only`...
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminMonitorBase.java
##########
@@ -151,4 +162,34 @@ public int getPendingNodeCount() {
public Queue<DatanodeDescriptor> getPendingNodes() {
return pendingNodes;
}
+
+ /**
+ * If node "is dead while in Decommission In Progress", it cannot be
decommissioned
+ * until it becomes healthy again. If there are more pendingNodes than can
be tracked
+ * & some unhealthy tracked nodes, then re-queue the unhealthy tracked nodes
+ * to avoid blocking decommissioning of healthy nodes.
+ *
+ * @param unhealthyDns The unhealthy datanodes which may be re-queued
+ * @param numDecommissioningNodes The total number of nodes being
decommissioned
+ * @return List of unhealthy nodes to be re-queued
Review comment:
nit: `@return Stream of unhealthy....`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
