aajisaka commented on a change in pull request #3675:
URL: https://github.com/apache/hadoop/pull/3675#discussion_r770299279
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
##########
@@ -270,12 +274,34 @@ private void check() {
// an invalid state.
LOG.warn("DatanodeAdminMonitor caught exception when processing node "
+ "{}.", dn, e);
- pendingNodes.add(dn);
- toRemove.add(dn);
+ toRequeue.add(dn);
Review comment:
I suppose reverting this change seems more simple because it can remove
`toRequeue` variable.
```suggestion
getPendingNodes.add(dn);
toRemove.add(dn);
```
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminMonitorBase.java
##########
@@ -151,4 +163,34 @@ public int getPendingNodeCount() {
public Queue<DatanodeDescriptor> getPendingNodes() {
return pendingNodes;
}
+
+ /**
+ * If node "is dead while in Decommission In Progress", it cannot be
decommissioned
+ * until it becomes healthy again. If there are more pendingNodes than can
be tracked
+ * & some unhealthy tracked nodes, then re-queue the unhealthy tracked nodes
+ * to avoid blocking decommissioning of healthy nodes.
+ *
+ * @param unhealthyDns The unhealthy datanodes which may be re-queued
+ * @param numDecommissioningNodes The total number of nodes being
decommissioned
+ * @return List of unhealthy nodes to be re-queued
+ */
+ List<DatanodeDescriptor> identifyUnhealthyNodesToRequeue(
+ final List<DatanodeDescriptor> unhealthyDns, int
numDecommissioningNodes) {
Review comment:
Using Stream instead of List for this method may simplify the code and
reduce the conversion between List and Stream. What do you think?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminMonitorBase.java
##########
@@ -35,12 +38,21 @@
public abstract class DatanodeAdminMonitorBase
implements DatanodeAdminMonitorInterface, Configurable {
+ /**
+ * Sort by lastUpdate time descending order, such that unhealthy
+ * nodes are de-prioritized given they cannot be decommissioned.
+ */
+ public static final Comparator<DatanodeDescriptor>
PENDING_NODES_QUEUE_COMPARATOR =
+ (dn1, dn2) -> Long.compare(dn2.getLastUpdate(), dn1.getLastUpdate());
+
protected BlockManager blockManager;
protected Namesystem namesystem;
protected DatanodeAdminManager dnAdmin;
protected Configuration conf;
- protected final Queue<DatanodeDescriptor> pendingNodes = new ArrayDeque<>();
+ private final PriorityQueue<DatanodeDescriptor> pendingNodes = new
PriorityQueue<>(
+
DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT,
Review comment:
I think we don't need to explicitly set the initial capacity. If the max
concurrent tracked nodes is set to lower value than the default, the capacity
will be too large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
