[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387557#comment-14387557 ] Allen Wittenauer commented on HADOOP-6857: -- This is still sitting in the changes.txt file for branch-2 and probably 2.7 FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387626#comment-14387626 ] Konstantin Shvachko commented on HADOOP-6857: - Because it was committed to branch-2 and 2.7. As discussed earlier there is bug fix, which went everywhere, and the print format change, which was reverted from branch 2 by HADOOP-11666. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347032#comment-14347032 ] Hudson commented on HADOOP-6857: SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346705#comment-14346705 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346715#comment-14346715 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/856/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346920#comment-14346920 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346929#comment-14346929 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346995#comment-14346995 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345855#comment-14345855 ] Konstantin Shvachko commented on HADOOP-6857: - Sorry missed your comment 2 days ago. Let me look at it. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345882#comment-14345882 ] Konstantin Shvachko commented on HADOOP-6857: - So there is a bug fix here and the output format change. The latter is incompatible for branch 2. I think we should revert the format change, but leave the bug fix. Byron, could you please confirm. Akira, does it make sense to you? FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346450#comment-14346450 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-trunk-Commit #7251 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7251/]) Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073) * hadoop-common-project/hadoop-common/CHANGES.txt FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 3.0.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345992#comment-14345992 ] Akira AJISAKA commented on HADOOP-6857: --- Filed HADOOP-11666. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345962#comment-14345962 ] Akira AJISAKA commented on HADOOP-6857: --- Thanks [~Byron Wong] for the patch. By the way, can we revert the incompatible change in a separate jira? Release note automatically collects information from JIRA (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/releasenotes.html), so I'd like to make them as follows: * HADOOP-6857: Fix version - 3.0.0 and mark as incompatible change. * A separate jira that reverts incompatible change: Fix version - 2.7.0 (commit branch-2 only) FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345898#comment-14345898 ] Byron Wong commented on HADOOP-6857: Yea, I agree. The bug fix should remain, but the output changes can be reverted. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345896#comment-14345896 ] Akira AJISAKA commented on HADOOP-6857: --- Make sense. We should leave the bug fix. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342741#comment-14342741 ] Akira AJISAKA commented on HADOOP-6857: --- This change is incompatible since it changes the output of the CLI command. (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI) Hi [~shv], should we revert this change from branch-2? FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184465#comment-14184465 ] Hudson commented on HADOOP-6857: SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/724/]) HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184496#comment-14184496 ] Hudson commented on HADOOP-6857: SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/]) HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184509#comment-14184509 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/]) HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Fix For: 2.7.0 Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184234#comment-14184234 ] Hudson commented on HADOOP-6857: FAILURE: Integrated in Hadoop-trunk-Commit #6345 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6345/]) HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml * hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183273#comment-14183273
]
Hadoop QA commented on HADOOP-6857:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
against trunk revision e2be333.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4948//console
This message is automatically generated.
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch,
show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183765#comment-14183765
]
Hadoop QA commented on HADOOP-6857:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
against trunk revision a52eb4b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//console
This message is automatically generated.
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
Attachments: HADOOP-6857.patch, HADOOP-6857.patch, HADOOP-6857.patch,
show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182244#comment-14182244 ] Konstantin Shvachko commented on HADOOP-6857: - This looks good. +1 from me. The test indeed failes on trunk and succeeds with the patch. Let's check with Jenkins. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182394#comment-14182394
]
Hadoop QA commented on HADOOP-6857:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676380/HADOOP-6857.patch
against trunk revision db45f04.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFSForHA
org.apache.hadoop.cli.TestHDFSCLI
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//console
This message is automatically generated.
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
Attachments: HADOOP-6857.patch, HADOOP-6857.patch,
show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182453#comment-14182453 ] Konstantin Shvachko commented on HADOOP-6857: - Looks like format change of du reporting broke TestHDFSCLI. Not sure about TestWebHDFSForHA. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175741#comment-14175741 ] Konstantin Shvachko commented on HADOOP-6857: - *Scenario 1* looks rather unusual, but again this is how quotas are counted: When you modify a file in a snapshot, then it is counted twice for quotas. Even if it doesn't actually store axtra blocks. If people want to change this, it should be a different jira related to qoutas. I don't think anything needs to be fixed in this regard here. *Scenario 2* indeed gives inconsistent results for _du_ and _dus_, because _du_ resolves path to point to the current tree rather than to the snapshot contents. This doesn't look right to me. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172644#comment-14172644 ] Konstantin Shvachko commented on HADOOP-6857: - Byron, Could you post some examples (printouts) in support of your use case. ??if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10?? This seems to be consistent with how quotas are calculated. DU should report numbers consistent with the quotas. We can change diskSpaceConsumed() if people want more precise usage info both for du and quotas, but at this point it will be a change in semantics. Let's keep it the current way. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172771#comment-14172771
]
Byron Wong commented on HADOOP-6857:
*Scenario 2*: we still have snapshottable directory /test with same file
a. We then create a fresh snapshot ss1. We then run {{hadoop fs -rm
-skipTrash /test/a}}.
{{hadoop fs -du /test}} gives an empty output, as expected.
{{hadoop fs -du -s /test}} outputs:
{code}
41 123 /test
{code}
which makes sense, given that we know about the existence of the snapshot.
However, when we run {{hadoop fs -du -s /test/.snapshot/ss1}}, we get:
{code}
0 0 /test/.snapshot/ss1
{code}
This is inconsistent with the numbers we get when we run {{hadoop fs -du
/test/.snapshot/ss1}}:
{code}
41 123 /test/.snapshot/ss1/a
{code}
Upon further investigation, we see that running {{hadoop fs -du -s
/test/.snapshot/anySnapshot}} gives us the information about the current state
of the real directory. This means that {{hadoop fs -du -s
/test/.snapshot/anySnapshot}} is equivalent to running {{hadoop fs -du /test/}}
and adding the numbers up, which is non-intuitive.
For example, let's add a 2 byte, 3 replication file /test/1 (/test/a is still
deleted). Now {{hadoop fs -du -s /test/.snapshot/ss1}} gives us:
{code}
2 6 /test/.snapshot/ss1
{code}
whereas the results of {{hadoop fs -du /test/.snapshot/ss1}} remains the same:
{code}
41 123 /test/.snapshot/ss1/a
{code}
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
Attachments: HADOOP-6857.patch, show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171831#comment-14171831 ] Byron Wong commented on HADOOP-6857: In the case when a directory /D and snapshot S are in the exact same state (e.g. a fresh snapshot has been made), everything works fine, meaning the sum of the disk consumed numbers reported by -du /D equals the disk consumed number reported by -du -s /D. When /D and S start deviating (files getting renamed, deleted, etc.), the disk consumed calculation will take the lastFileSize within the snapshots, find the maximum replication factor for that file within the snapshots, multiply the 2 together, and increment disk consumed by that number, which inflates the total disk consumed calculation, so -du -s /D the sum of numbers in -du /D. I'd also like to point out that this implementation only takes replication factor of a file into account, even if that replication factor is greater than number of data nodes, which further inflates the -du calculation. For example, if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10, and report that number. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170322#comment-14170322 ] Konstantin Shvachko commented on HADOOP-6857: - Byron, could you please clarify how -du reporting is different in the presence of snapshots? FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Assignee: Byron Wong Attachments: HADOOP-6857.patch, show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909693#action_12909693 ] Maxim Veksler commented on HADOOP-6857: --- Is there an easy cli method of obtaining this information? If not then this can be a welcome feature in several use cases (Like file server based on HDFS) FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909788#action_12909788
]
Eli Collins commented on HADOOP-6857:
-
I agree with Koji, seeing raw disk space usage should be easy to get from the
CLI, you shouldn't have to enable a quota on a directory to see it, eg see
below that nothing indicates raw disk usage:
{code}
~ $ hadoop fs -mkdir dir
~ $ hadoop fs -put f3mb dir
~ $ hadoop fs -dus dir
hdfs://haus01.sf.cloudera.com:10020/user/eli/dir3145728
~ $ hadoop fs -count -q dir
none infnone inf1
13145728 hdfs://haus01.sf.cloudera.com:10020/user/eli/dir
{code}
It also sounds like the original issue here was that disk space usage should
not have been added to count if it was already available in dus. Even if we
added raw disk usage to count, it's wonky that a user can logical disk usage
with du but has to switch over to a separate count command to get raw disk
usage. Why not make all disk usage available from a single command?
Sound reasonable Nicholas?
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Fix For: 0.22.0
Attachments: show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909368#action_12909368 ] Aaron T. Myers commented on HADOOP-6857: bq. We already have fs -count path which counts bytes including replications. Is it good enough? fs -count path is indeed sufficient. Feel free to close this ticket. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909472#action_12909472
]
Eli Collins commented on HADOOP-6857:
-
Why do we have separate dus and count commands? They seem to duplicate
each other. Since the CLI needs to be backwards compatible not suggesting we
remove one, but perhaps we should update the hdfs_shell docs to make it clear
that they display the same info if there are not significant differences.
{code}
~ $ hadoop fs -dus /user/eli
hdfs://haus01.sf.cloudera.com:10020/user/eli86183666860
{code}
{code}
~ $ hadoop fs -count /user/eli
7 5186183666860
hdfs://haus01.sf.cloudera.com:10020/user/eli
~ $
{code}
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Fix For: 0.22.0
Attachments: show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909478#action_12909478 ] Tsz Wo (Nicholas), SZE commented on HADOOP-6857: fs -count was introduced for counting name objects. The disk space column was added later on. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909502#action_12909502
]
Koji Noguchi commented on HADOOP-6857:
--
A little confused. I thought fs -count shows hdfs usage same as fs -du at
the thrid column.
{noformat}
[knoguchi ~]$ hadoop dfs -dus /user/knoguchi
hdfs://abc-nn1.com/user/knoguchi 2603203340273
[knoguchi ~]$ hadoop dfs -count /user/knoguchi
158020624 2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$
{noformat}
If quota is enabled on that dir and -q is passed, it would show the remaining
raw space available.
{noformat}
[knoguchi ~]$ hadoop dfs -count -q /user/knoguchi
5 27796 13194139533312 5384528402193 1580
20624 2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$
{noformat}
You can get the raw space usage then. (quota - raw\_remaining).
However *this is only if you have quota enabled on that particular dir*.
FsShell should report raw disk usage including replication factor
-
Key: HADOOP-6857
URL: https://issues.apache.org/jira/browse/HADOOP-6857
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Alex Kozlov
Fix For: 0.22.0
Attachments: show-space-consumed.txt
Currently FsShell report HDFS usage with hadoop fs -dus path command.
Since replication level is per file level, it would be nice to add raw disk
usage including the replication factor (maybe hadoop fs -dus -raw path?).
This will allow to assess resource usage more accurately. -- Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890070#action_12890070 ] Tsz Wo (Nicholas), SZE commented on HADOOP-6857: We already have fs -count path which counts bytes including replications. Is it good enough? FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Alex Kozlov Fix For: 0.22.0 Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6857) FsShell should report raw disk usage including replication factor
[ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889481#action_12889481 ] Hadoop QA commented on HADOOP-6857: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449749/show-space-consumed.txt against trunk revision 964993. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/console This message is automatically generated. FsShell should report raw disk usage including replication factor - Key: HADOOP-6857 URL: https://issues.apache.org/jira/browse/HADOOP-6857 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 0.20.2 Reporter: Alex Kozlov Attachments: show-space-consumed.txt Currently FsShell report HDFS usage with hadoop fs -dus path command. Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe hadoop fs -dus -raw path?). This will allow to assess resource usage more accurately. -- Alex K -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
