dzcxzl created HDFS-18038:
-----------------------------
Summary: Add a NameNode metric for the number of low redundancy
blocks on decommissioning DataNodes
Key: HDFS-18038
URL: https://issues.apache.org/jira/browse/HDFS-18038
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: dzcxzl
The NameNode exposes the number of decommissioning DataNodes as the
NumDecommissioningDataNodes metric, but the number of blocks that still have to
be replicated before those DataNodes can be decommissioned is only available
per node, inside the DecomNodes JSON string of the NameNodeInfo MXBean
(underReplicatedBlocks). Monitoring systems that only collect numeric metrics
cannot plot or alert on the decommission progress.
With DatanodeAdminDefaultMonitor the progress can be roughly inferred from
LowRedundancyBlocks, because all the blocks of a decommissioning DataNode are
added to the reconstruction queue at once. With DatanodeAdminBackoffMonitor
this no longer works: the number of blocks added to the reconstruction queue is
limited by dfs.namenode.decommission.backoff.monitor.pending.limit, so
LowRedundancyBlocks does not reflect the remaining work.
This issue proposes a new FSNamesystem metric,
DecommissioningLowRedundancyBlocks, which is the sum of the low redundancy
blocks tracked for each DataNode in decommissioning state, i.e. the total of
the per node underReplicatedBlocks value reported in DecomNodes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]