[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906766#comment-16906766
]
Hudson commented on HDFS-14491:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17110 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17110/])
HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. (weichiu:
rev c13ec7ab666fc4878174a7cd952ca93941ae7c05)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906752#comment-16906752
]
Wei-Chiu Chuang commented on HDFS-14491:
+1 gotcha. Makes sense to me now.
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904414#comment-16904414
]
Siyao Meng commented on HDFS-14491:
---
Also as shown in your snippet of *InvalidateBlocks#add*, this operation should
be performed for *each* datanode that has the replica. Hence the
*PendingDeletionBlocks* metric does include all replicas/EC strips.
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904413#comment-16904413
]
Siyao Meng commented on HDFS-14491:
---
[~jojochuang]
The metric is retrieved via *FSNameSystem#getPendingDeletionBlocks*
[here|https://github.com/apache/hadoop/blob/b964b81f8509ba6cd938bc36f3acb5e3112b7ca2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L4918]
Then it calls
{code:title=BlockManager#getPendingDeletionBlocksCount}
/** Used by metrics */
public long getPendingDeletionBlocksCount() {
return invalidateBlocks.numBlocks();
}
{code}
If we look into how *invalidateBlocks* is changed, we can see in
*BlockManager#addToInvalidates*
{code:title=BlockManager#addToInvalidates}
/**
* Adds block to list of blocks which will be invalidated on all its
* datanodes.
*/
private void addToInvalidates(BlockInfo storedBlock) {
if (!isPopulatingReplQueues()) {
return;
}
StringBuilder datanodes = blockLog.isDebugEnabled()
? new StringBuilder() : null;
for (DatanodeStorageInfo storage : blocksMap.getStorages(storedBlock)) {
if (storage.getState() != State.NORMAL) {
continue;
}
final DatanodeDescriptor node = storage.getDatanodeDescriptor();
final Block b = getBlockOnStorage(storedBlock, storage);
if (b != null) {
invalidateBlocks.add(b, node, false);
if (datanodes != null) {
datanodes.append(node).append(" ");
}
}
}
if (datanodes != null && datanodes.length() != 0) {
blockLog.debug("BLOCK* addToInvalidates: {} {}", storedBlock, datanodes);
}
}
{code}
So we can see that *invalidateBlocks* is indeed incremented for each block on
each datanode. CMIIW
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864789#comment-16864789
]
Wei-Chiu Chuang commented on HDFS-14491:
I'm not sure about this one.
Looking at the code, we doesn't count replica numbers. It counts the unique
blocks.
{code}
/**
* Add a block to the block collection which will be
* invalidated on the specified datanode.
*/
synchronized void add(final Block block, final DatanodeInfo datanode,
final boolean log) {
LightWeightHashSet set = getBlocksSet(datanode, block);
if (set == null) {
set = new LightWeightHashSet<>();
putBlocksSet(datanode, block, set);
}
if (set.add(block)) {
if (blockIdManager.isStripedBlock(block)) {
numECBlocks.increment();
} else {
numBlocks.increment();
}
if (log) {
NameNode.blockStateChangeLog.debug("BLOCK* {}: add {} to {}",
getClass().getSimpleName(), block, datanode);
}
}
}
{code}
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844684#comment-16844684
]
Hadoop QA commented on HDFS-14491:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969249/HDFS-14491.001.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux a89d6d250501 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1cb2eb0 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26810/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
