[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469812#comment-15469812
]
Akira Ajisaka commented on HDFS-10778:
--
+1, I verified the option in both oiv and oiv_legacy commands. I'll commit this
tomorrow if there are no objections.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch, HDFS-10778.004.patch, HDFS-10778.005.patch,
> HDFS-10778.006.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466730#comment-15466730
]
Hadoop QA commented on HDFS-10778:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827126/HDFS-10778.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux df13f6a83387 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 07650bc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16640/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16640/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
>
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466139#comment-15466139
]
Akira Ajisaka commented on HDFS-10778:
--
Thanks [~linyiqun] for updating the patch. I tried your patch and get the
following output:
{noformat}
[centos@ip-172-31-21-203 conf]$ hdfs oiv -p FileDistribution -format -step 30
-maxSize 300 -i /hadoop/dfs/name/current/fsimage_307
Processed 0 inodes.
Size Range NumFiles
(0 B, 30 B] 2
(270 B, 300 B] 32
totalFiles = 34
totalDirectories = 13
totalBlocks = 34
totalSpace = 288198
maxFileSize = 160321
{noformat}
Actually maxFileSize is 160321 but the output says the file is in {{(270 B, 300
B\]}}. Would you fix it to output (270B, maxFileSize]?
If a fsimage includes empty files, the output is as follows:
{noformat}
Size Range NumFiles
(0 B, 0 B] 1
(0 B, 30 B] 2
{noformat}
I'm thinking \[0 B, 0 B\] is better than (0 B, 0 B\].
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch, HDFS-10778.004.patch, HDFS-10778.005.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464324#comment-15464324
]
Hadoop QA commented on HDFS-10778:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
21s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827039/HDFS-10778.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ccf1e568ed17 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 07650bc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16633/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16633/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16633/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464115#comment-15464115
]
Akira Ajisaka commented on HDFS-10778:
--
The patch looks good to me. Would you rebase the patch?
Sorry for the late response.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch, HDFS-10778.004.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457441#comment-15457441
]
Yiqun Lin commented on HDFS-10778:
--
Hi [~ajisakaa], can we go ahead for this JIRA, the latest patch can satisfy
with you? Thanks a lot if you have time to take a look for this.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch, HDFS-10778.004.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430699#comment-15430699
]
Hadoop QA commented on HDFS-10778:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m
0s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824818/HDFS-10778.004.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 80f2cd6b9572 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430587#comment-15430587
]
Yiqun Lin commented on HDFS-10778:
--
Attach a new patch to fix checkstyle warnings.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430348#comment-15430348
]
Hadoop QA commented on HDFS-10778:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824789/HDFS-10778.003.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3453b4065648 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430228#comment-15430228
]
Yiqun Lin commented on HDFS-10778:
--
Thanks [~ajisakaa] for the review.
{quote}
Would you file another jira for this issue and split the patch?
{quote}
Done. I created the HDFS-10783 for tracking this and posted a simple patch for
that.
Update the patch of this jira.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430058#comment-15430058
]
Hadoop QA commented on HDFS-10778:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
40s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824761/HDFS-10778.002.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ac792a567872 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16493/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16493/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16493/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430025#comment-15430025
]
Akira Ajisaka commented on HDFS-10778:
--
Thanks [~linyiqun] for updating the patch!
bq. The option '-h' is already used for -help in hdfs oiv command. So it seems
we would the other option, now I use a new option -format to instead of that.
Agreed.
bq. I found another bug when I tested the new option in class
OfflineImageViewer, it is missing the following code in method
OfflineImageViewer#buildOptions:
Nice catch. Would you file another jira for this issue and split the patch? The
bug is similar to HDFS-5866.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430007#comment-15430007
]
Yiqun Lin commented on HDFS-10778:
--
Hi, [~ajisakaa], now I working on this jira and I found some other problems
here.
{quote}
so would you add a new option to optimize the output? '-h' is good for me.
{quote}
The option '-h' is already used for {{-help}} in hdfs oiv command. So it seems
we would the other option, now I use a new option {{-format}} to instead of
that.
I found another bug when I tested the new option in class
{{OfflineImageViewer}}, it is missing the following code in method
{{OfflineImageViewer#buildOptions}}:
{code}
options.addOption("maxSize", true, "");
options.addOption("step", true, "");
{code}
Then leads the {{ParseException}} being threw when doing the
{{parser.parse(options, args)}}.
{code}
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
-maxSize
{code}
The test output in my local env:
{code}
with -format option:
Size Range NumFiles
(0 B, 8 B] 12
totalFiles = 12
totalDirectories = 9
totalBlocks = 12
totalSpace = 12
maxFileSize = 1
without -format option:
SizeNumFiles
8 12
totalFiles = 12
totalDirectories = 9
totalBlocks = 12
totalSpace = 12
maxFileSize = 1
{code}
Finally, attach a new patch for this, thanks for the review.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428006#comment-15428006
]
Yiqun Lin commented on HDFS-10778:
--
Thanks [~ajisakaa] for the quick response. I will attach a new patch to address
your comment next week.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427774#comment-15427774
]
Akira Ajisaka commented on HDFS-10778:
--
Thanks [~linyiqun] for the patch. The improved output looks great.
Changing the output format of CLI is incompatible, so would you add a new
option to optimize the output? '-h' is good for me. In addition, we need to
document the option.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427638#comment-15427638
]
Hadoop QA commented on HDFS-10778:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 16 unchanged - 1 fixed = 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824495/HDFS-10778.001.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9157fe755a7f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dbcaf99 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16479/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16479/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427564#comment-15427564
]
Yiqun Lin commented on HDFS-10778:
--
Softly ping [~ajisakaa], I think this is also a improvement for users to use
hdfs oiv like HDFS-10691. Could you take a look for this, looking forward to
seeing your response.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427560#comment-15427560
]
Yiqun Lin commented on HDFS-10778:
--
Some test output result in my test cluster:
{code}
Size Range NumFiles
(0 B, 0 B] 16663
(0, 1 M]778416
(1 M, 2 M] 35125
(2 M, 3 M] 13278
(3 M, 4 M] 10458
...
(126 M, 127 M] 1589
(127 M, 128 M] 24971
totalFiles = 960947
totalDirectories = 30795
totalBlocks = 996233
totalSpace = 35208724040079
maxFileSize = 2593079816
{code}
Attach a initial patch.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
