dzcxzl created HDFS-18038:
-----------------------------

             Summary: Add a NameNode metric for the number of low redundancy 
blocks on decommissioning DataNodes
                 Key: HDFS-18038
                 URL: https://issues.apache.org/jira/browse/HDFS-18038
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
            Reporter: dzcxzl


The NameNode exposes the number of decommissioning DataNodes as the 
NumDecommissioningDataNodes metric, but the number of blocks that still have to 
be replicated before those DataNodes can be decommissioned is only available 
per node, inside the DecomNodes JSON string of the NameNodeInfo MXBean 
(underReplicatedBlocks). Monitoring systems that only collect numeric metrics 
cannot plot or alert on the decommission progress.

With DatanodeAdminDefaultMonitor the progress can be roughly inferred from 
LowRedundancyBlocks, because all the blocks of a decommissioning DataNode are 
added to the reconstruction queue at once. With DatanodeAdminBackoffMonitor 
this no longer works: the number of blocks added to the reconstruction queue is 
limited by dfs.namenode.decommission.backoff.monitor.pending.limit, so 
LowRedundancyBlocks does not reflect the remaining work.

This issue proposes a new FSNamesystem metric, 
DecommissioningLowRedundancyBlocks, which is the sum of the low redundancy 
blocks tracked for each DataNode in decommissioning state, i.e. the total of 
the per node underReplicatedBlocks value reported in DecomNodes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to