[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632027#comment-16632027 ] Takanobu Asanuma commented on HDFS-8196: Since the latest patch focuses only on enabled ec policies, I'd like to change the title of this jira. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kitti Nanasi >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, > HDFS-8196.06.patch, HDFS-8196.07.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632021#comment-16632021 ] Takanobu Asanuma commented on HDFS-8196: The failed tests passed in my local computer. Will commit it soon. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kitti Nanasi >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, > HDFS-8196.06.patch, HDFS-8196.07.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631803#comment-16631803
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestPersistBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941606/HDFS-8196.07.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6d30bf7b1f56 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7093afd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25162/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25162/testReport/ |
| Max. process+thread count | 3230 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631655#comment-16631655 ] Takanobu Asanuma commented on HDFS-8196: +1, pending jenkins. Seems there was something wrong with the last build. I reran Jenkins. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kitti Nanasi >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, > HDFS-8196.06.patch, HDFS-8196.07.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631139#comment-16631139 ] Kitti Nanasi commented on HDFS-8196: Thanks for the comments [~tasanuma0829], I agree with them and fixed them in patch v007. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kitti Nanasi >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, > HDFS-8196.06.patch, HDFS-8196.07.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628759#comment-16628759
]
Takanobu Asanuma commented on HDFS-8196:
Thanks for working on this, [~knanasi]. Some minor comments
* For {{EnabledEcPolicies}}, {{ECBlockGroupsMBean}} might be better than
{{FSNamesystemMBean}}.
* I'd prefer that {{Block Deletion Start Time}} and {{Last Checkpoint Time}}
are consecutive rows.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kitti Nanasi
>Priority: Major
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch,
> HDFS-8196.06.patch, Screen Shot 2017-02-06 at 22.30.40.png, Screen Shot
> 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614108#comment-16614108
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
222 unchanged - 0 fixed = 223 total (was 222) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
54s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939591/HDFS-8196.06.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e4bd3011a4b2 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613872#comment-16613872 ] Kitti Nanasi commented on HDFS-8196: Thanks for reviewing [~xiaochen]! Most of it was intentional, like instead of defining a new bean, I put the metric in FSNamesystem, and I didn't keep the default ec policy metrics, but it would be good to show the enabled policies in the dfshealth, so I will provide a patch fixing that. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, Screen Shot > 2017-02-06 at 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen > Shot 2017-02-14 at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613010#comment-16613010 ] Xiao Chen commented on HDFS-8196: - Thanks for picking up the ball here [~knanasi]. It seems the html/js changes are removed in the latest rev. Is that intentional? > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, HDFS-8196.05.patch, Screen Shot > 2017-02-06 at 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen > Shot 2017-02-14 at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611582#comment-16611582
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
223 unchanged - 0 fixed = 224 total (was 223) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
33s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939341/HDFS-8196.05.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b46d11b84a2e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b2432d2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs |
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611439#comment-16611439 ] Kitti Nanasi commented on HDFS-8196: [~lewuathe], I will upload a patch for this soon, let me know if you would like to make a change. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594887#comment-16594887
]
genericqa commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-8196 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24898/console |
| Powered by | Apache Yetus 0.9.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Major
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594875#comment-16594875 ] Kitti Nanasi commented on HDFS-8196: I think it would be great to have the ErasureCodingPolicyManagerMBean that you implemented here, [~lewuathe]. Do you plan to continue working on this? The BlockManager#getTotalStripedBlockGroups and BlockManager#getTotalStripedBlockSize metrics are not needed anymore, because HDFS-12573 already implements those (in BlockManager#getTotalECBlockGroups), but the rest of the patch would be very useful to have. Also it would be nice to extend TestNameNodeMXBean#testEcPolicyManagerMetrics with a check for getActivePolicies. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407022#comment-16407022
]
genericqa commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-8196 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23583/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Major
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252036#comment-16252036
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-8196 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22085/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149867#comment-16149867
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-8196 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20955/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149864#comment-16149864 ] Andrew Wang commented on HDFS-8196: --- Ping, think we can get this patch revved and in? > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052248#comment-16052248
]
Manoj Govindassamy commented on HDFS-8196:
--
[~lewuathe],
{{ClientProtocol#getECBlockGroupsStats()}} and
{{ClientProtocol#getECBlocksStats()}} are modelled similar to
{{ClientProtocol#getStats()}}. That is, the stats cover only the problamatic
block counts. As of now, the total block replica blocks and EC blockgroups
count are not available in any of these APIs. We need to extend these stats to
cover the new requirements. Sorry for the delayed response.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012464#comment-16012464
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HDFS-8196 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19451/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012409#comment-16012409
]
Kai Sasaki commented on HDFS-8196:
--
[~manojg] Thank you so much and I'm so sorry for late response. I have one
question regarding HDFS-10999 and this JIRA.
I was trying to fetch the total number of ec blocks the the total size of ec
blocks. But {{ECBlockGroupsStats}} does not include these metrics.
Can we get the necessary information from {{ECBlockGroupsStats}}? Please let me
know if I overlook something.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at
> 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14
> at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966327#comment-15966327 ] Manoj Govindassamy commented on HDFS-8196: -- [~lewuathe], posted a patch in HDFS-10999 for addressing the EC and Replicated block accounting in a thread safe manner. Let me know if those block accounting can be made use for this jira issue. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886785#comment-15886785 ] Andrew Wang commented on HDFS-8196: --- Looks like the 04 patch doesn't address my comments about not iterating the whole block map to calculate these counts, also thread safety. I think [~manojg] is planning to add these counts in the context of HDFS-10999, so maybe we revisit this JIRA after that. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, > HDFS-8196.03.patch, HDFS-8196.04.patch, Screen Shot 2017-02-06 at > 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 > at 22.43.57.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884811#comment-15884811
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests | org.apache.hadoop.tools.TestJMXGet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854772/HDFS-8196.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux adccffe0ff91 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 05391c1 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18447/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18447/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18447/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
>
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870478#comment-15870478
]
Andrew Wang commented on HDFS-8196:
---
Thanks for the rev [~lewuathe], did another pass, one nit and one more major
thing I missed before (apologies):
Nit:
{code}
Total EC Striped Block Groups Size
(bytes){StripedBlocksTotalSize}
{code}
I think this should read:
{code}
Total EC Striped Block Groups
Size{StripedBlocksTotalSize|fmt_bytes}
{code}
Major:
{code}
4032 public long getTotalStripedBlocks() {
4033return StreamSupport.stream(blocksMap.getBlocks().spliterator(),
false)
4034.filter(BlockInfo::isStriped)
4035.count();
4036 }
4037
4038 public long getTotalStripedBlockSize() {
4039return StreamSupport.stream(blocksMap.getBlocks().spliterator(),
false)
4040.filter(BlockInfo::isStriped)
4041.mapToLong(BlockInfo::getNumBytes)
4042.sum();
4043 }
{code}
Looks like these new methods iterate the entire block map to calculate the
totals, which is inefficient. It also looks not thread safe.
Instead, what do you think about maintaining these counts in {{BlocksMap}}?
Another small nit, maybe we rename {{getTotalStripedBlocks}} to
{{getTotalStripedBlockGroups}} for clarity?
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, Screen Shot 2017-02-06 at 22.30.40.png, Screen Shot
> 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865963#comment-15865963
]
Hadoop QA commented on HDFS-8196:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 372 unchanged - 0 fixed = 390 total (was 372) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests | org.apache.hadoop.tools.TestJMXGet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12852569/HDFS-8196.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 349ca84a1344 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4164a20 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18371/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18371/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18371/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18371/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure Coding related information on NameNode UI
>
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865808#comment-15865808
]
Kai Sasaki commented on HDFS-8196:
--
Thanks for review!
{quote}
Should we leave per-policy statistics as an enhancement for later? I think this
is okay since most of the time we'll only have one EC policy in use on a
cluster.
{quote}
I agree. We can implement it in follow on tasks. In terms of showing erasure
coding policy per directory, we can implement in HDFS-11394.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch,
> HDFS-8196.03.patch, Screen Shot 2017-02-06 at 22.30.40.png, Screen Shot
> 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864656#comment-15864656
]
Andrew Wang commented on HDFS-8196:
---
I like this new rev, some nits:
* Can we add spaces between the entries in the list of active policies? Looks a
bit cramped right now.
* For the total size and cell size, I think we should use {{fmt_bytes}} like
the other byte numbers on dfshealth.html.
Should we leave per-policy statistics as an enhancement for later? I think this
is okay since most of the time we'll only have one EC policy in use on a
cluster.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, Screen Shot
> 2017-02-06 at 22.30.40.png, Screen Shot 2017-02-12 at 20.21.42.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861436#comment-15861436
]
Takanobu Asanuma commented on HDFS-8196:
Thanks for your thoughts, [~lewuathe]!
bq. After contiguous block is implemented, we can add another field to show the
metrics of the ones. I think showing the metrics by block group type (striped,
contiguous) is helpful.
I understand it. For more clear, how about using {{EC Striped}} for now?
{noformat}
Total EC Striped Block Groups
Total EC Striped Block Groups Size
{noformat}
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at
> 22.30.40.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861284#comment-15861284
]
Kai Sasaki commented on HDFS-8196:
--
[~tasanuma0829] Thanks for comments!
{quote}
How about summarizing one row?
{quote}
Looks good. I'll try.
{quote}
We plan to implement EC + Contiguous in HDFS-8030.
{quote}
After contiguous block is implemented, we can add another field to show the
metrics of the ones. I think showing the metrics by block group type (striped,
contiguous) is helpful. But if {{fsck}} shows two type of block group metrics
as one field, we can consolidate them as {{Erasure-Coded}} metrics as you said.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at
> 22.30.40.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859498#comment-15859498
]
Takanobu Asanuma commented on HDFS-8196:
Thanks for the patch, [~lewuathe], and thanks for reviewing, [~andrew.wang]!
Some minor comments:
* I feel it might be redundant to separate four rows for the default schema.
How about summarizing one row?
{noformat}
RS-DEFAUT-6-3-64K (Data Blocks: 6, Parity Blocks: 3, Cell Size: 65536 bytes)
{noformat}
* I think {{Block Groups}} is more fit for the metrics of the total amount.
{noformat}
Total Striped Blocks -> Total Striped Block Groups
Total Striped Blocks Size -> Total Striped Block Groups Size
{noformat}
* About the above, is the word of {{Striped}} appropriate here? We plan to
implement {{EC}} + {{Contiguous}} in HDFS-8030. If we summarize them, the word
of {{Erasure-Coded}} would be better. The result of fsck may be a reference for
this task.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at
> 22.30.40.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859467#comment-15859467 ] Kai Sasaki commented on HDFS-8196: -- [~andrew.wang] Thanks for quick review! I'll update accordingly. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at > 22.30.40.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857027#comment-15857027
]
Andrew Wang commented on HDFS-8196:
---
Nice work here [~lewuathe]! Some small review comments:
* The EC Cell Size is a property of the policy, so it'd be nice to move it down
one row, and display it like the Num Data/Parity Blocks names, i.e. {{Cell Size
(RS-DEFAULT-6-3-64k)}}. I think it'd be better to also put the units with the
value, like {{65536 bytes}}.
* Per the above, we should fetch the cell size from the default policy, not
from the constant in ECPM, and name the getter {{getDefaultPolicyCellSize}} for
consistency.
* Some sanity tests for the new MBean metrics in ECPM would be good
* Slightly odd whitespace here:
{noformat}
211 MBeans.register("NameNode",
212 "ErasureCodingPolicyManager", bean);
{noformat}
* Noticed that ECPM is abbreviated {{ecm}} in dfshealth.html and .js, would
prefer {{ecpm}} for clarity.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at
> 22.30.40.png
>
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855916#comment-15855916 ] Kai Sasaki commented on HDFS-8196: -- I uploaded a patch which shows - Default erasure coding policy information (cell size, num data/parity blocks) - Total striped blocks - Total striped blocks size Information of each directory is shown through WebHDFS currently. So we can modify WebHDFS to expose erasure coding policy information by directory in HDFS-11394 separately. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > Attachments: HDFS-8196.01.patch, Screen Shot 2017-02-06 at > 22.30.40.png > > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834632#comment-15834632 ] Kai Sasaki commented on HDFS-8196: -- The displayed information can be In browsing utility - EC policy of erasure coded directory In overview - Total erasure coded file size, file count, blocks count per EC policy - Available EC policy in a cluster > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830899#comment-15830899
]
Kai Sasaki commented on HDFS-8196:
--
[~andrew.wang] Thanks. I can do.
{quote}
My first thoughts are some counts on how much data in the cluster is erasure
coded, in terms capacity, blocks, files, average # blocks / file
{quote}
Yes. In addition to these capacity metrics, I think showing available codecs,
configs are useful for admins. I'll list up the information should be shown.
> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
>
> NameNode WebUI shows EC related information and metrics.
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830831#comment-15830831 ] Andrew Wang commented on HDFS-8196: --- [~lewuathe] if you still have interest please go ahead, I think Manoj is already working on a few other JIRAs. Could you also describe what kinds of info you plan to add to the WebUI? My first thoughts are some counts on how much data in the cluster is erasure coded, in terms capacity, blocks, files, average # blocks / file, and all of this per EC policy. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829522#comment-15829522 ] Kai Sasaki commented on HDFS-8196: -- [~andrew.wang] Sorry, HDFS-7674 which blocks this issue was out of my rador. I can check HDFS-7674 updates and into integrate this but if [~manojg] is interested in of course please picking it up. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI
[ https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829501#comment-15829501 ] Andrew Wang commented on HDFS-8196: --- [~lewuathe] are you still working on this JIRA? Otherwise, I think [~manojg] might be interested in picking this up. > Erasure Coding related information on NameNode UI > - > > Key: HDFS-8196 > URL: https://issues.apache.org/jira/browse/HDFS-8196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-7285 >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have > > NameNode WebUI shows EC related information and metrics. > This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
