[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904413#comment-16904413
]
Siyao Meng commented on HDFS-14491:
-----------------------------------
[~jojochuang]
The metric is retrieved via *FSNameSystem#getPendingDeletionBlocks*
[here|https://github.com/apache/hadoop/blob/b964b81f8509ba6cd938bc36f3acb5e3112b7ca2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L4918]
Then it calls
{code:title=BlockManager#getPendingDeletionBlocksCount}
/** Used by metrics */
public long getPendingDeletionBlocksCount() {
return invalidateBlocks.numBlocks();
}
{code}
If we look into how *invalidateBlocks* is changed, we can see in
*BlockManager#addToInvalidates*
{code:title=BlockManager#addToInvalidates}
/**
* Adds block to list of blocks which will be invalidated on all its
* datanodes.
*/
private void addToInvalidates(BlockInfo storedBlock) {
if (!isPopulatingReplQueues()) {
return;
}
StringBuilder datanodes = blockLog.isDebugEnabled()
? new StringBuilder() : null;
for (DatanodeStorageInfo storage : blocksMap.getStorages(storedBlock)) {
if (storage.getState() != State.NORMAL) {
continue;
}
final DatanodeDescriptor node = storage.getDatanodeDescriptor();
final Block b = getBlockOnStorage(storedBlock, storage);
if (b != null) {
invalidateBlocks.add(b, node, false);
if (datanodes != null) {
datanodes.append(node).append(" ");
}
}
}
if (datanodes != null && datanodes.length() != 0) {
blockLog.debug("BLOCK* addToInvalidates: {} {}", storedBlock, datanodes);
}
}
{code}
So we can see that *invalidateBlocks* is indeed incremented for each block on
each datanode. CMIIW
> More Clarity on Namenode UI Around Blocks and Replicas
> ------------------------------------------------------
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Alan Jackoway
> Assignee: Siyao Meng
> Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 40000000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
