[jira] [Updated] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12018: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Fix For: HDFS-7240 > > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch, HDFS-12018-HDFS-7240.003.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12045: Status: Patch Available (was: Open) > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: diskbalancer >Reporter: steven-wugang >Assignee: Anu Engineer > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064263#comment-16064263 ] Weiwei Yang commented on HDFS-12018: I just committed v3 patch to the feature branch, thanks [~vagarychen] for the contribution, thanks for [~xyao] for the review. > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch, HDFS-12018-HDFS-7240.003.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12011) Add a new load balancing volume choosing policy
[
https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064259#comment-16064259
]
Hadoop QA commented on HDFS-12011:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 498 unchanged - 0 fixed = 499 total (was 498) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874604/HADOOP-12011.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 129bb4832789 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2b87faf |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20056/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20056/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20056/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20056/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064256#comment-16064256 ] Weiwei Yang commented on HDFS-12018: The jenkins result seems good, +1 to the v3 patch. I am going to commit this shortly. > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch, HDFS-12018-HDFS-7240.003.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12028) Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
[
https://issues.apache.org/jira/browse/HDFS-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12028:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
> Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
> ---
>
> Key: HDFS-12028
> URL: https://issues.apache.org/jira/browse/HDFS-12028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Fix For: HDFS-7240
>
> Attachments: HDFS-12028-HDFS-7240.001.patch
>
>
> Currently when you run CLI "hdfs oz ...", there always a noisy slf4j biding
> issue log output. This ticket is opened to removed it.
> {code}
> xyao$ hdfs oz
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/hdfs/lib/logback-classic-1.0.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064250#comment-16064250 ] Anu Engineer edited comment on HDFS-12045 at 6/27/17 4:41 AM: -- +1, pending jenkins. Thanks for taking care of this oversight. I will commit this as soon as I have a jenkins run. Btw, Can I add you to the contributors list, so I can assign this JIRA to you ? was (Author: anu): +1, pending jenkins. Thanks for take care of this oversight. I will commit this as soon as I have a jenkins run. Btw, Can I add you to the contributors list, so I can assign this JIRA to you ? > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: diskbalancer >Reporter: steven-wugang >Assignee: Anu Engineer > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12028) Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
[
https://issues.apache.org/jira/browse/HDFS-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064251#comment-16064251
]
Weiwei Yang commented on HDFS-12028:
Just committed to the feature branch, thanks for [~vagarychen] to get this
fixed, thanks for [~xyao] and [~anu]'s comments and reviews.
> Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
> ---
>
> Key: HDFS-12028
> URL: https://issues.apache.org/jira/browse/HDFS-12028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-12028-HDFS-7240.001.patch
>
>
> Currently when you run CLI "hdfs oz ...", there always a noisy slf4j biding
> issue log output. This ticket is opened to removed it.
> {code}
> xyao$ hdfs oz
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/hdfs/lib/logback-classic-1.0.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12045: Component/s: diskbalancer > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement > Components: diskbalancer >Reporter: steven-wugang >Assignee: Anu Engineer > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064250#comment-16064250 ] Anu Engineer commented on HDFS-12045: - +1, pending jenkins. Thanks for take care of this oversight. I will commit this as soon as I have a jenkins run. Btw, Can I add you to the contributors list, so I can assign this JIRA to you ? > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: steven-wugang >Assignee: Anu Engineer > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-12045: --- Assignee: Anu Engineer > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: steven-wugang >Assignee: Anu Engineer > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12028) Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
[
https://issues.apache.org/jira/browse/HDFS-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064246#comment-16064246
]
Anu Engineer commented on HDFS-12028:
-
+1 [~vagarychen] Thanks for taking care of this. I independently tested this
now in a cluster and I can confirm it works well. Thank you for taking care of
this. [~cheersyang] Thanks for the review and the commit.
> Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
> ---
>
> Key: HDFS-12028
> URL: https://issues.apache.org/jira/browse/HDFS-12028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-12028-HDFS-7240.001.patch
>
>
> Currently when you run CLI "hdfs oz ...", there always a noisy slf4j biding
> issue log output. This ticket is opened to removed it.
> {code}
> xyao$ hdfs oz
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/hdfs/lib/logback-classic-1.0.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12028) Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
[
https://issues.apache.org/jira/browse/HDFS-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064245#comment-16064245
]
Weiwei Yang commented on HDFS-12028:
Hi [~vagarychen]
You are right. I just did a clean and rebuilt, now the warnings are gone, you
patch works. Thank you. I am going to commit this patch soon. Thank you for
fixing this, it looks really better now :).
Thank you.
> Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
> ---
>
> Key: HDFS-12028
> URL: https://issues.apache.org/jira/browse/HDFS-12028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-12028-HDFS-7240.001.patch
>
>
> Currently when you run CLI "hdfs oz ...", there always a noisy slf4j biding
> issue log output. This ticket is opened to removed it.
> {code}
> xyao$ hdfs oz
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/hdfs/lib/logback-classic-1.0.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11870) Add CLI cmd to enable/disable an erasure code policy
[
https://issues.apache.org/jira/browse/HDFS-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064238#comment-16064238
]
Hadoop QA commented on HDFS-11870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11870 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874600/HDFS-11870.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux f44de52bdcda 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2b87faf |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20055/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20055/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20055/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064230#comment-16064230
]
Yiqun Lin commented on HDFS-12040:
--
LGTM, +1. Thanks [~kihwal] for reminding me.
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-12040.001.patch, HDFS-12040-branch-2.001.patch
>
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12025) Modify help instructions of Plan command and Execute command about DiskBalancer
[
https://issues.apache.org/jira/browse/HDFS-12025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064223#comment-16064223
]
Hadoop QA commented on HDFS-12025:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874598/HDFS-12025.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d74b36021f96 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2b87faf |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20054/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20054/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Modify help instructions of Plan command and Execute command about
> DiskBalancer
> ---
>
> Key: HDFS-12025
> URL: https://issues.apache.org/jira/browse/HDFS-12025
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: steven-wugang
> Attachments: HDFS-12025.002.patch
>
[jira] [Commented] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064215#comment-16064215
]
Hadoop QA commented on HDFS-12040:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874596/HDFS-12040.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 70eba3c26394 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2b87faf |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20053/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20053/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-12040.001.patch, HDFS-12040-branch-2.001.patch
>
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
>
[jira] [Updated] (HDFS-12046) Hadoop CRC implementation using Intel ISAL library
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-12046: - Summary: Hadoop CRC implementation using Intel ISAL library (was: Hadoop CRC implementation using Intel ISAL-CRC library) > Hadoop CRC implementation using Intel ISAL library > -- > > Key: HDFS-12046 > URL: https://issues.apache.org/jira/browse/HDFS-12046 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: luhuichun >Assignee: luhuichun > > Intel ISA-L open source library provides set of highly optimized functions > for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. > Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and > added the necessary building options support for Hadoop. For Hadoop CRC, we > recently explored more, developing a Hadoop CRC using Intel ISA-L, performing > a test on Broadwell and Skylake servers, comparing the performance against > Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance > gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using > the ISA-L library, the extra advantage is it’s already optimized when we > upgrade to new servers and Hadoop developers don’t have to maintain their own > bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12046) Hadoop CRC implementation using Intel ISAL-CRC library
luhuichun created HDFS-12046: Summary: Hadoop CRC implementation using Intel ISAL-CRC library Key: HDFS-12046 URL: https://issues.apache.org/jira/browse/HDFS-12046 Project: Hadoop HDFS Issue Type: Improvement Reporter: luhuichun Assignee: luhuichun Intel ISA-L open source library provides set of highly optimized functions for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and added the necessary building options support for Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L, performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have to maintain their own bunch of ASM codes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12011) Add a new load balancing volume choosing policy
[ https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chencan updated HDFS-12011: --- Attachment: HADOOP-12011.002.patch > Add a new load balancing volume choosing policy > > > Key: HDFS-12011 > URL: https://issues.apache.org/jira/browse/HDFS-12011 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12011.002.patch, HADOOP-12011.patch > > > There are two types of volume choosing policies when choose a volume > inner a datanode to write in a datablock : RoundRobinVolumeChoosingPolicy and > AvailableSpaceVolumeChoosingPolicy.This two policies do not take into account > the fsvolume's load. We can add a new load balancing volume choosing policy, > using existing reference in FsVolumeImpl. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
[ https://issues.apache.org/jira/browse/HDFS-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12045: - Attachment: HDFS-12045.patch > Add log when Diskbalancer volume is transient storage type > -- > > Key: HDFS-12045 > URL: https://issues.apache.org/jira/browse/HDFS-12045 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: steven-wugang > Attachments: HDFS-12045.patch > > > When dskbalancer volume is transient storage type,should add log to explain > that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12011) Add a new load balancing volume choosing policy
[ https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chencan updated HDFS-12011: --- Attachment: (was: HADOOP-12011.003.patch) > Add a new load balancing volume choosing policy > > > Key: HDFS-12011 > URL: https://issues.apache.org/jira/browse/HDFS-12011 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12011.patch > > > There are two types of volume choosing policies when choose a volume > inner a datanode to write in a datablock : RoundRobinVolumeChoosingPolicy and > AvailableSpaceVolumeChoosingPolicy.This two policies do not take into account > the fsvolume's load. We can add a new load balancing volume choosing policy, > using existing reference in FsVolumeImpl. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12011) Add a new load balancing volume choosing policy
[ https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chencan updated HDFS-12011: --- Attachment: (was: HADOOP-12011.002.path) > Add a new load balancing volume choosing policy > > > Key: HDFS-12011 > URL: https://issues.apache.org/jira/browse/HDFS-12011 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12011.patch > > > There are two types of volume choosing policies when choose a volume > inner a datanode to write in a datablock : RoundRobinVolumeChoosingPolicy and > AvailableSpaceVolumeChoosingPolicy.This two policies do not take into account > the fsvolume's load. We can add a new load balancing volume choosing policy, > using existing reference in FsVolumeImpl. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12045) Add log when Diskbalancer volume is transient storage type
steven-wugang created HDFS-12045: Summary: Add log when Diskbalancer volume is transient storage type Key: HDFS-12045 URL: https://issues.apache.org/jira/browse/HDFS-12045 Project: Hadoop HDFS Issue Type: Improvement Reporter: steven-wugang When dskbalancer volume is transient storage type,should add log to explain that unable to support transient storage type。 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11870) Add CLI cmd to enable/disable an erasure code policy
[
https://issues.apache.org/jira/browse/HDFS-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064147#comment-16064147
]
lufei commented on HDFS-11870:
--
Thank you very much for your careful check. I have Modified in patch
{HDFS-11870.005.patch} according to your suggestion.
> Add CLI cmd to enable/disable an erasure code policy
>
>
> Key: HDFS-11870
> URL: https://issues.apache.org/jira/browse/HDFS-11870
> Project: Hadoop HDFS
> Issue Type: Task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: SammiChen
>Assignee: lufei
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11870.001.patch, HDFS-11870.002.patch,
> HDFS-11870.003.patch, HDFS-11870.004.patch, HDFS-11870.005.patch
>
>
> This task is to develop a CLI command to help user enable or disable and
> existing erasure coding policy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11870) Add CLI cmd to enable/disable an erasure code policy
[ https://issues.apache.org/jira/browse/HDFS-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-11870: - Attachment: HDFS-11870.005.patch > Add CLI cmd to enable/disable an erasure code policy > > > Key: HDFS-11870 > URL: https://issues.apache.org/jira/browse/HDFS-11870 > Project: Hadoop HDFS > Issue Type: Task > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: SammiChen >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11870.001.patch, HDFS-11870.002.patch, > HDFS-11870.003.patch, HDFS-11870.004.patch, HDFS-11870.005.patch > > > This task is to develop a CLI command to help user enable or disable and > existing erasure coding policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12025) Modify help instructions of Plan command and Execute command about DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-12025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12025: - Status: Patch Available (was: Open) > Modify help instructions of Plan command and Execute command about > DiskBalancer > --- > > Key: HDFS-12025 > URL: https://issues.apache.org/jira/browse/HDFS-12025 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang > Attachments: HDFS-12025.002.patch > > > There are some inaccurate descriptions in help descriptions about PlanCommand > and ExecuteCommand of DiskBalancer. > For example, in ExecuteCommand help description, "Execute command runs a > submits a plan for execution on the given data node" should be modify > modified to "Execute command runs a submitted plan for execution on the given > data node". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12025) Modify help instructions of Plan command and Execute command about DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-12025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12025: - Attachment: (was: HDFS-12025.001.patch) > Modify help instructions of Plan command and Execute command about > DiskBalancer > --- > > Key: HDFS-12025 > URL: https://issues.apache.org/jira/browse/HDFS-12025 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang > Attachments: HDFS-12025.002.patch > > > There are some inaccurate descriptions in help descriptions about PlanCommand > and ExecuteCommand of DiskBalancer. > For example, in ExecuteCommand help description, "Execute command runs a > submits a plan for execution on the given data node" should be modify > modified to "Execute command runs a submitted plan for execution on the given > data node". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12025) Modify help instructions of Plan command and Execute command about DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-12025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12025: - Attachment: HDFS-12025.002.patch > Modify help instructions of Plan command and Execute command about > DiskBalancer > --- > > Key: HDFS-12025 > URL: https://issues.apache.org/jira/browse/HDFS-12025 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang > Attachments: HDFS-12025.002.patch > > > There are some inaccurate descriptions in help descriptions about PlanCommand > and ExecuteCommand of DiskBalancer. > For example, in ExecuteCommand help description, "Execute command runs a > submits a plan for execution on the given data node" should be modify > modified to "Execute command runs a submitted plan for execution on the given > data node". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12025) Modify help instructions of Plan command and Execute command about DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-12025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12025: - Status: Open (was: Patch Available) > Modify help instructions of Plan command and Execute command about > DiskBalancer > --- > > Key: HDFS-12025 > URL: https://issues.apache.org/jira/browse/HDFS-12025 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang > Attachments: HDFS-12025.002.patch > > > There are some inaccurate descriptions in help descriptions about PlanCommand > and ExecuteCommand of DiskBalancer. > For example, in ExecuteCommand help description, "Execute command runs a > submits a plan for execution on the given data node" should be modify > modified to "Execute command runs a submitted plan for execution on the given > data node". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hu xiaodong updated HDFS-12040:
---
Attachment: HDFS-12040.001.patch
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-12040.001.patch, HDFS-12040-branch-2.001.patch
>
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hu xiaodong updated HDFS-12040:
---
Affects Version/s: 3.0.0-alpha3
Status: Patch Available (was: Open)
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-12040.001.patch, HDFS-12040-branch-2.001.patch
>
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hu xiaodong updated HDFS-12040:
---
Attachment: HDFS-12040-branch-2.001.patch
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-12040-branch-2.001.patch
>
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hu xiaodong reassigned HDFS-12040:
--
Assignee: hu xiaodong
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064056#comment-16064056
]
Hadoop QA commented on HDFS-6984:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
43s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 38s{color}
| {color:red} root generated 79 new + 823 unchanged - 0 fixed = 902 total (was
823) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 12 new + 783 unchanged
- 22 fixed = 795 total (was 805) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
41s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-hdfs-httpfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-6984 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874572/HDFS-6984.014.patch |
| Optional Tests | asflicense xml compile javac javadoc mvninstall
mvnsite unit findbugs checkstyle cc |
| uname | Linux 508b980bd7f1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a9d3412 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| javac |
[jira] [Created] (HDFS-12044) Mismatch between BlockManager#maxReplicatioStreams and ErasureCodingWorker.stripedReconstructionPool pool size causes slow and burst recovery.
Lei (Eddy) Xu created HDFS-12044:
Summary: Mismatch between BlockManager#maxReplicatioStreams and
ErasureCodingWorker.stripedReconstructionPool pool size causes slow and burst
recovery.
Key: HDFS-12044
URL: https://issues.apache.org/jira/browse/HDFS-12044
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0-alpha3
Reporter: Lei (Eddy) Xu
{{ErasureCodingWorker#stripedReconstructionPool}} is with {{corePoolSize=2}}
and {{maxPoolSize=8}} as default. And it rejects more tasks if the queue is
full.
When {{BlockManager#maxReplicationStream}} is larger than
{{ErasureCodingWorker#stripedReconstructionPool#corePoolSize/maxPoolSize}}, for
example, {{maxReplicationStream=20}} and {{corePoolSize=2 , maxPoolSize=8}}.
Meanwhile, NN sends up to {{maxTransfer}} reconstruction tasks to DN for each
heartbeat, and it is calculated in {{FSNamesystem}}:
{code}
final int maxTransfer = blockManager.getMaxReplicationStreams() -
xmitsInProgress;
{code}
However, at any giving time, {{{ErasureCodingWorker#stripedReconstructionPool}}
takes 2 {{xmitInProcess}}. So for each heartbeat in 3s, NN will send about
{{20-2 = 18}} reconstruction tasks to the DN, and DN throw away most of them if
there were 8 tasks in the queue already. So NN needs to take longer to
re-consider these blocks were under-replicated to schedule new tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12042) Reduce memory used by snapshot diff data structures
[
https://issues.apache.org/jira/browse/HDFS-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064036#comment-16064036
]
Hadoop QA commented on HDFS-12042:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 17 unchanged - 1 fixed = 23 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874579/HDFS-12042.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 394d0c52e9c1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 144753e |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20052/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-12033) DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks.
[
https://issues.apache.org/jira/browse/HDFS-12033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063946#comment-16063946
]
Hudson commented on HDFS-12033:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11927 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11927/])
HDFS-12033. DatanodeManager picking EC recovery tasks should also (lei: rev
144753e87f4a9daa51200be05ff2bb760bf38169)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
> DatanodeManager picking EC recovery tasks should also consider the number of
> regular replication tasks.
> ---
>
> Key: HDFS-12033
> URL: https://issues.apache.org/jira/browse/HDFS-12033
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-12033.00.patch, HDFS-12033.01.patch
>
>
> In {{DatanodeManager#handleHeartbeat}}, it choose both pending replication
> list and pending EC list to up to {{maxTransfers}} items.
> It should only send {{maxTransfers}} tasks combined to DN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12033) DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks.
[
https://issues.apache.org/jira/browse/HDFS-12033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12033:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
Status: Resolved (was: Patch Available)
Thanks for reviews, [~andrew.wang]
Committed to trunk.
> DatanodeManager picking EC recovery tasks should also consider the number of
> regular replication tasks.
> ---
>
> Key: HDFS-12033
> URL: https://issues.apache.org/jira/browse/HDFS-12033
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-12033.00.patch, HDFS-12033.01.patch
>
>
> In {{DatanodeManager#handleHeartbeat}}, it choose both pending replication
> list and pending EC list to up to {{maxTransfers}} items.
> It should only send {{maxTransfers}} tasks combined to DN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12042) Reduce memory used by snapshot diff data structures
[
https://issues.apache.org/jira/browse/HDFS-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12042:
--
Status: Patch Available (was: Open)
> Reduce memory used by snapshot diff data structures
> ---
>
> Key: HDFS-12042
> URL: https://issues.apache.org/jira/browse/HDFS-12042
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12042.01.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Some of that memory
> is wasted due to suboptimal data structures, such as empty or under-populated
> ArrayLists, etc. Analyzing one heap dump with jxray (www.jxray.com), we
> observed the following problems with data structures:
> {code}
> 9. BAD COLLECTIONS
> Total collections: 99,707,902 Bad collections: 88,799,760 Overhead:
> 9,063,898K (18.2%)
> Top bad collections:
> Ovhd Problem Num objs Type
> -
> 3,056,014K (6.1%) small 29435572 j.u.ArrayList
> 2,641,373K (5.3%) 1-elem 21837906 j.u.ArrayList
> 864,215K (1.7%) 1-elem 5291813 j.u.TreeSet
> 808,456K (1.6%) 1-elem 3045847 j.u.HashMap
> 602,470K (1.2%) empty 18549109 j.u.ArrayList
> 441,563K (0.9%) empty 4356975 j.u.TreeSet
> 373,088K (0.7%) empty 5297007 j.u.HashMap
> 270,324K (0.5%) small 931394 j.u.HashMap
> {code}
> The data structures created by HDFS code that suffer from the above problems
> are, in particular:
> {code}
> 4,228,182K (8.5%): j.u.ArrayList: 19412263 of small 2,111,087K (4.2%),
> 12932408 of 1-elem 1,717,585K (3.4%), 12784310 of empty 399,509K (0.8%)
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs
> <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> {code}
> and
> {code}
> 575,557K (1.2%): j.u.ArrayList: 4363271 of 1-elem 409,056K (0.8%), 2439001
> of small 166,482K (0.3%)
> <-- org.apache.hadoop.hdfs.server.namenode.INodeDirectory.children <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.util.LightWeightGSet.entries <--
> org.apache.hadoop.hdfs.server.namenode.INodeMap.map <--
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
> <-- org.apache.hadoop.util.Daemon.target <--
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
> <-- org.apache.hadoop.util.Daemon.target <-- j.l.Thread[] <--
> j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java Static:
> org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> {code}
> There are several different reference chains that all lead to
> FileDiffList.diffs or INodeDirectory.children. The total percentage of memory
> wasted by these data structures in the analyzed dump is about 12%. By
> creating these lists lazily and/or with capacity that better matches their
> actual size, we should be able to reclaim a significant part of these 12%.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12042) Reduce memory used by snapshot diff data structures
[
https://issues.apache.org/jira/browse/HDFS-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12042:
--
Attachment: HDFS-12042.01.patch
> Reduce memory used by snapshot diff data structures
> ---
>
> Key: HDFS-12042
> URL: https://issues.apache.org/jira/browse/HDFS-12042
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12042.01.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Some of that memory
> is wasted due to suboptimal data structures, such as empty or under-populated
> ArrayLists, etc. Analyzing one heap dump with jxray (www.jxray.com), we
> observed the following problems with data structures:
> {code}
> 9. BAD COLLECTIONS
> Total collections: 99,707,902 Bad collections: 88,799,760 Overhead:
> 9,063,898K (18.2%)
> Top bad collections:
> Ovhd Problem Num objs Type
> -
> 3,056,014K (6.1%) small 29435572 j.u.ArrayList
> 2,641,373K (5.3%) 1-elem 21837906 j.u.ArrayList
> 864,215K (1.7%) 1-elem 5291813 j.u.TreeSet
> 808,456K (1.6%) 1-elem 3045847 j.u.HashMap
> 602,470K (1.2%) empty 18549109 j.u.ArrayList
> 441,563K (0.9%) empty 4356975 j.u.TreeSet
> 373,088K (0.7%) empty 5297007 j.u.HashMap
> 270,324K (0.5%) small 931394 j.u.HashMap
> {code}
> The data structures created by HDFS code that suffer from the above problems
> are, in particular:
> {code}
> 4,228,182K (8.5%): j.u.ArrayList: 19412263 of small 2,111,087K (4.2%),
> 12932408 of 1-elem 1,717,585K (3.4%), 12784310 of empty 399,509K (0.8%)
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs
> <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> {code}
> and
> {code}
> 575,557K (1.2%): j.u.ArrayList: 4363271 of 1-elem 409,056K (0.8%), 2439001
> of small 166,482K (0.3%)
> <-- org.apache.hadoop.hdfs.server.namenode.INodeDirectory.children <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.util.LightWeightGSet.entries <--
> org.apache.hadoop.hdfs.server.namenode.INodeMap.map <--
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
> <-- org.apache.hadoop.util.Daemon.target <--
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
> <-- org.apache.hadoop.util.Daemon.target <-- j.l.Thread[] <--
> j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java Static:
> org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> {code}
> There are several different reference chains that all lead to
> FileDiffList.diffs or INodeDirectory.children. The total percentage of memory
> wasted by these data structures in the analyzed dump is about 12%. By
> creating these lists lazily and/or with capacity that better matches their
> actual size, we should be able to reclaim a significant part of these 12%.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12033) DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks.
[
https://issues.apache.org/jira/browse/HDFS-12033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063900#comment-16063900
]
Andrew Wang commented on HDFS-12033:
+1 LGTM, thanks Eddy!
> DatanodeManager picking EC recovery tasks should also consider the number of
> regular replication tasks.
> ---
>
> Key: HDFS-12033
> URL: https://issues.apache.org/jira/browse/HDFS-12033
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12033.00.patch, HDFS-12033.01.patch
>
>
> In {{DatanodeManager#handleHeartbeat}}, it choose both pending replication
> list and pending EC list to up to {{maxTransfers}} items.
> It should only send {{maxTransfers}} tasks combined to DN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-6984:
Attachment: HDFS-6984.014.patch
Rebased patch
> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch,
> HDFS-6984.003.patch, HDFS-6984.004.patch, HDFS-6984.005.patch,
> HDFS-6984.006.patch, HDFS-6984.007.patch, HDFS-6984.008.patch,
> HDFS-6984.009.patch, HDFS-6984.010.patch, HDFS-6984.011.patch,
> HDFS-6984.012.patch, HDFS-6984.013.patch, HDFS-6984.014.patch,
> HDFS-6984.nowritable.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this
> to serialize it and send it over the wire. But in Hadoop 2 and later, we
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this
> information. The protobuf form is preferable, since it allows us to add new
> fields in a backwards-compatible way. Another issue is that already a lot of
> subclasses of FileStatus don't override the Writable methods of the
> superclass, breaking the interface contract that read(status.write) should be
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so
> that we don't have to deal with these issues. It's probably too late to do
> this in Hadoop 2, since user code may be relying on the existing FileStatus
> serialization there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12033) DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks.
[
https://issues.apache.org/jira/browse/HDFS-12033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063812#comment-16063812
]
Hadoop QA commented on HDFS-12033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874544/HDFS-12033.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f9534e7efdd4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2c367b4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20050/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20050/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DatanodeManager picking EC recovery tasks should also consider the number of
> regular replication tasks.
> ---
>
> Key: HDFS-12033
> URL: https://issues.apache.org/jira/browse/HDFS-12033
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12033.00.patch, HDFS-12033.01.patch
>
>
> In
[jira] [Created] (HDFS-12043) Add counters for block re-replication
Chen Liang created HDFS-12043: - Summary: Add counters for block re-replication Key: HDFS-12043 URL: https://issues.apache.org/jira/browse/HDFS-12043 Project: Hadoop HDFS Issue Type: Bug Reporter: Chen Liang Assignee: Chen Liang We occasionally see that the under-replicated block count is not going down quickly enough. We've made at least one fix to speed up block replications (HDFS-9205) but we need better insight into the current state and activity of the block re-replication logic. For example, we need to understand whether is it because re-replication is not making forward progress at all, or is it because new under-replicated blocks are being added faster. We should include additional metrics: # Cumulative number of blocks that were successfully replicated. # Cumulative number of re-replications that timed out. # Cumulative number of blocks that were dequeued for re-replication but not scheduled e.g. because they were invalid, or under-construction or replication was postponed. The growth rate of of the above metrics will make it clear whether block replication is making forward progress and if not then provide potential clues about why it is stalled. Thanks [~arpitagarwal] for the offline discussions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11993) Add log info when connect to datanode socket address failed
[
https://issues.apache.org/jira/browse/HDFS-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063761#comment-16063761
]
Hudson commented on HDFS-11993:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11926 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11926/])
HDFS-11993. Add log info when connect to datanode socket address failed.
(raviprak: rev a9d3412b4ce40f5ab5a18756ede7e0606b653171)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
> Add log info when connect to datanode socket address failed
> ---
>
> Key: HDFS-11993
> URL: https://issues.apache.org/jira/browse/HDFS-11993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: chencan
>Assignee: chencan
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-11993.002.patch, HADOOP-11993.patch
>
>
> In function BlockSeekTo, when connect faild to datanode socket address,log as
> follow:
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
> + ", add to deadNodes and continue. " + ex, ex);
> add block info may be more explicit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[
https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063755#comment-16063755
]
Andrew Wang commented on HDFS-11882:
I added the nice-to-have label to it, thanks Zhe.
> Client fails if acknowledged size is greater than bytes sent
>
>
> Key: HDFS-11882
> URL: https://issues.apache.org/jira/browse/HDFS-11882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11882.01.patch
>
>
> Some tests of erasure coding fails by the following exception. The following
> test was removed by HDFS-11823, however, this type of error can happen in
> real cluster.
> {noformat}
> Running
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure)
> Time elapsed: 38.831 sec <<< ERROR!
> java.lang.IllegalStateException: null
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11787) After HDFS-11515, -du still throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/HDFS-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-11787. Resolution: Duplicate Fix Version/s: (was: 2.8.2) (was: 3.0.0-alpha4) (was: 2.9.0) Target Version/s: 2.8.1, 3.0.0-beta1 (was: 3.0.0-beta1, 2.8.1) Duping to HDFS-11515 since I don't think we need this for changelog purposes. > After HDFS-11515, -du still throws ConcurrentModificationException > -- > > Key: HDFS-11787 > URL: https://issues.apache.org/jira/browse/HDFS-11787 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots, tools >Affects Versions: 3.0.0-alpha4, 2.8.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > > I ran a modified NameNode that was patched against HDFS-11515 on a production > cluster fsimage, and I am still seeing ConcurrentModificationException. > It seems that there are corner cases not convered by HDFS-11515. File this > jira to discuss how to proceed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-11515) -du throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-11515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HDFS-11515:
I think we can simplify the JIRA history since this change never made it into a
release, re-opening.
> -du throws ConcurrentModificationException
> --
>
> Key: HDFS-11515
> URL: https://issues.apache.org/jira/browse/HDFS-11515
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, shell
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Istvan Fajth
> Attachments: HDFS-11515.001.patch, HDFS-11515.002.patch,
> HDFS-11515.003.patch, HDFS-11515.004.patch, HDFS-11515.test.patch
>
>
> HDFS-10797 fixed a disk summary (-du) bug, but it introduced a new bug.
> The bug can be reproduced running the following commands:
> {noformat}
> bash-4.1$ hdfs dfs -mkdir /tmp/d0
> bash-4.1$ hdfs dfsadmin -allowSnapshot /tmp/d0
> Allowing snaphot on /tmp/d0 succeeded
> bash-4.1$ hdfs dfs -touchz /tmp/d0/f4
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1
> bash-4.1$ hdfs dfs -createSnapshot /tmp/d0 s1
> Created snapshot /tmp/d0/.snapshot/s1
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d2
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d3
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d2/d4
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d3/d5
> bash-4.1$ hdfs dfs -createSnapshot /tmp/d0 s2
> Created snapshot /tmp/d0/.snapshot/s2
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d2/d4
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d2
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d3/d5
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d3
> bash-4.1$ hdfs dfs -du -h /tmp/d0
> du: java.util.ConcurrentModificationException
> 0 0 /tmp/d0/f4
> {noformat}
> A ConcurrentModificationException forced du to terminate abruptly.
> Correspondingly, NameNode log has the following error:
> {noformat}
> 2017-03-08 14:32:17,673 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getContentSumma
> ry from 10.0.0.198:49957 Call#2 Retry#0
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.hdfs.server.namenode.ContentSummaryComputationContext.tallyDeletedSnapshottedINodes(ContentSummaryComputationContext.java:209)
> at
> org.apache.hadoop.hdfs.server.namenode.INode.computeAndConvertContentSummary(INode.java:507)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.getContentSummary(FSDirectory.java:2302)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:4535)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1087)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getContentSummary(AuthorizationProviderProxyClientProtocol.java:5
> 63)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.jav
> a:873)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> The bug is due to a improper use of HashSet, not concurrent operations.
> Basically, a HashSet can not be updated while an iterator is traversing it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-11787) After HDFS-11515, -du still throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/HDFS-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-11787: > After HDFS-11515, -du still throws ConcurrentModificationException > -- > > Key: HDFS-11787 > URL: https://issues.apache.org/jira/browse/HDFS-11787 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots, tools >Affects Versions: 3.0.0-alpha4, 2.8.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > > I ran a modified NameNode that was patched against HDFS-11515 on a production > cluster fsimage, and I am still seeing ConcurrentModificationException. > It seems that there are corner cases not convered by HDFS-11515. File this > jira to discuss how to proceed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11515) -du throws ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HDFS-11515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11515:
---
Fix Version/s: (was: 2.8.2)
(was: 3.0.0-alpha4)
(was: 2.9.0)
> -du throws ConcurrentModificationException
> --
>
> Key: HDFS-11515
> URL: https://issues.apache.org/jira/browse/HDFS-11515
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, shell
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Istvan Fajth
> Attachments: HDFS-11515.001.patch, HDFS-11515.002.patch,
> HDFS-11515.003.patch, HDFS-11515.004.patch, HDFS-11515.test.patch
>
>
> HDFS-10797 fixed a disk summary (-du) bug, but it introduced a new bug.
> The bug can be reproduced running the following commands:
> {noformat}
> bash-4.1$ hdfs dfs -mkdir /tmp/d0
> bash-4.1$ hdfs dfsadmin -allowSnapshot /tmp/d0
> Allowing snaphot on /tmp/d0 succeeded
> bash-4.1$ hdfs dfs -touchz /tmp/d0/f4
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1
> bash-4.1$ hdfs dfs -createSnapshot /tmp/d0 s1
> Created snapshot /tmp/d0/.snapshot/s1
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d2
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d3
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d2/d4
> bash-4.1$ hdfs dfs -mkdir /tmp/d0/d1/d3/d5
> bash-4.1$ hdfs dfs -createSnapshot /tmp/d0 s2
> Created snapshot /tmp/d0/.snapshot/s2
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d2/d4
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d2
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d3/d5
> bash-4.1$ hdfs dfs -rmdir /tmp/d0/d1/d3
> bash-4.1$ hdfs dfs -du -h /tmp/d0
> du: java.util.ConcurrentModificationException
> 0 0 /tmp/d0/f4
> {noformat}
> A ConcurrentModificationException forced du to terminate abruptly.
> Correspondingly, NameNode log has the following error:
> {noformat}
> 2017-03-08 14:32:17,673 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 4 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getContentSumma
> ry from 10.0.0.198:49957 Call#2 Retry#0
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.hdfs.server.namenode.ContentSummaryComputationContext.tallyDeletedSnapshottedINodes(ContentSummaryComputationContext.java:209)
> at
> org.apache.hadoop.hdfs.server.namenode.INode.computeAndConvertContentSummary(INode.java:507)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.getContentSummary(FSDirectory.java:2302)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getContentSummary(FSNamesystem.java:4535)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getContentSummary(NameNodeRpcServer.java:1087)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getContentSummary(AuthorizationProviderProxyClientProtocol.java:5
> 63)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getContentSummary(ClientNamenodeProtocolServerSideTranslatorPB.jav
> a:873)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
> {noformat}
> The bug is due to a improper use of HashSet, not concurrent operations.
> Basically, a HashSet can not be updated while an iterator is traversing it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way
[
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HDFS-11419:
My mistake, the commit message of a subtask included the JIRA # of this one.
Reopening.
> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up
> calling into {{chooseRandom}}, which will first find a random datanode by
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope,
> excludedNodes);{code}, then it checks whether that returned datanode
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
> chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes,
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after
> a random node is already returned. We've seen a case where a cluster has a
> large number of datanodes, while only a few satisfy the storage type
> condition. So, for the most part, this code blindly picks random datanodes
> that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is
> that, given a set of excluded nodes, it first finds a random datanodes, then
> if it is in excluded nodes set, try find another random nodes. So the more
> excluded nodes there are, the more likely a random node will be in the
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way
> that datanodes can be found more efficiently. There are currently two
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and
> when searching for a node, we take into account such field(s), and do not
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for
> one storage type, there is one tree subset that connects all the nodes with
> that type. And one search happens on only one such subset. So unexpected
> storage types are simply not in the search space.
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for
> pointing out a wrong statement (corrected now) in the description. Any
> further comments are more than welcome.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way
[
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11419:
---
Fix Version/s: (was: 3.0.0-alpha4)
> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up
> calling into {{chooseRandom}}, which will first find a random datanode by
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope,
> excludedNodes);{code}, then it checks whether that returned datanode
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
> chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes,
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after
> a random node is already returned. We've seen a case where a cluster has a
> large number of datanodes, while only a few satisfy the storage type
> condition. So, for the most part, this code blindly picks random datanodes
> that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is
> that, given a set of excluded nodes, it first finds a random datanodes, then
> if it is in excluded nodes set, try find another random nodes. So the more
> excluded nodes there are, the more likely a random node will be in the
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way
> that datanodes can be found more efficiently. There are currently two
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and
> when searching for a node, we take into account such field(s), and do not
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for
> one storage type, there is one tree subset that connects all the nodes with
> that type. And one search happens on only one such subset. So unexpected
> storage types are simply not in the search space.
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for
> pointing out a wrong statement (corrected now) in the description. Any
> further comments are more than welcome.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12042) Reduce memory used by snapshot diff data structures
[
https://issues.apache.org/jira/browse/HDFS-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12042:
--
Description:
When snapshot diff operation is performed in a NameNode that manages several
million HDFS files/directories, NN needs a lot of memory. Some of that memory
is wasted due to suboptimal data structures, such as empty or under-populated
ArrayLists, etc. Analyzing one heap dump with jxray (www.jxray.com), we
observed the following problems with data structures:
{code}
9. BAD COLLECTIONS
Total collections: 99,707,902 Bad collections: 88,799,760 Overhead:
9,063,898K (18.2%)
Top bad collections:
Ovhd Problem Num objs Type
-
3,056,014K (6.1%) small 29435572 j.u.ArrayList
2,641,373K (5.3%) 1-elem 21837906 j.u.ArrayList
864,215K (1.7%) 1-elem 5291813 j.u.TreeSet
808,456K (1.6%) 1-elem 3045847 j.u.HashMap
602,470K (1.2%) empty 18549109 j.u.ArrayList
441,563K (0.9%) empty 4356975 j.u.TreeSet
373,088K (0.7%) empty 5297007 j.u.HashMap
270,324K (0.5%) small 931394 j.u.HashMap
{code}
The data structures created by HDFS code that suffer from the above problems
are, in particular:
{code}
4,228,182K (8.5%): j.u.ArrayList: 19412263 of small 2,111,087K (4.2%),
12932408 of 1-elem 1,717,585K (3.4%), 12784310 of empty 399,509K (0.8%)
<-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
<-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
<-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
{code}
and
{code}
575,557K (1.2%): j.u.ArrayList: 4363271 of 1-elem 409,056K (0.8%), 2439001 of
small 166,482K (0.3%)
<-- org.apache.hadoop.hdfs.server.namenode.INodeDirectory.children <--
org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
org.apache.hadoop.util.LightWeightGSet.entries <--
org.apache.hadoop.hdfs.server.namenode.INodeMap.map <--
org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
<-- org.apache.hadoop.util.Daemon.target <--
org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
<-- org.apache.hadoop.util.Daemon.target <-- j.l.Thread[] <--
j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java Static:
org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
{code}
There are several different reference chains that all lead to
FileDiffList.diffs or INodeDirectory.children. The total percentage of memory
wasted by these data structures in the analyzed dump is about 12%. By creating
these lists lazily and/or with capacity that better matches their actual size,
we should be able to reclaim a significant part of these 12%.
was:
When snapshot diff operation is performed in a NameNode that manages several
million HDFS files/directories, NN needs a lot of memory. Some of that memory
is wasted due to suboptimal data structures, such as empty or under-populated
ArrayLists, etc. Analyzing one heap dump with jxray (www.jxray.com), we
observed the following problems with data structures:
{code}
9. BAD COLLECTIONS
Total collections: 99,707,902 Bad collections: 88,799,760 Overhead:
9,063,898K (18.2%)
Top bad collections:
Ovhd Problem Num objs Type
-
3,056,014K (6.1%) small 29435572 j.u.ArrayList
2,641,373K (5.3%) 1-elem 21837906 j.u.ArrayList
864,215K (1.7%) 1-elem 5291813 j.u.TreeSet
808,456K (1.6%) 1-elem 3045847 j.u.HashMap
602,470K (1.2%) empty 18549109 j.u.ArrayList
441,563K (0.9%) empty 4356975 j.u.TreeSet
373,088K (0.7%) empty 5297007 j.u.HashMap
270,324K (0.5%) small 931394
[jira] [Resolved] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way
[
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-11419.
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
Resolving as it looks like this was completed.
> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: 3.0.0-alpha4
>
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up
> calling into {{chooseRandom}}, which will first find a random datanode by
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope,
> excludedNodes);{code}, then it checks whether that returned datanode
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
> chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes,
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after
> a random node is already returned. We've seen a case where a cluster has a
> large number of datanodes, while only a few satisfy the storage type
> condition. So, for the most part, this code blindly picks random datanodes
> that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is
> that, given a set of excluded nodes, it first finds a random datanodes, then
> if it is in excluded nodes set, try find another random nodes. So the more
> excluded nodes there are, the more likely a random node will be in the
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way
> that datanodes can be found more efficiently. There are currently two
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and
> when searching for a node, we take into account such field(s), and do not
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for
> one storage type, there is one tree subset that connects all the nodes with
> that type. And one search happens on only one such subset. So unexpected
> storage types are simply not in the search space.
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for
> pointing out a wrong statement (corrected now) in the description. Any
> further comments are more than welcome.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11907) Add metric for time taken by NameNode resource check
[ https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11907: --- Fix Version/s: 3.0.0-alpha4 > Add metric for time taken by NameNode resource check > > > Key: HDFS-11907 > URL: https://issues.apache.org/jira/browse/HDFS-11907 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch, > HDFS-11907.003.patch, HDFS-11907.004.patch, HDFS-11907.005.patch, > HDFS-11907.006.patch, HDFS-11907.007.patch > > > Add a metric to measure the time taken by the NameNode Resource Check. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Closed] (HDFS-11993) Add log info when connect to datanode socket address failed
[
https://issues.apache.org/jira/browse/HDFS-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash closed HDFS-11993.
---
> Add log info when connect to datanode socket address failed
> ---
>
> Key: HDFS-11993
> URL: https://issues.apache.org/jira/browse/HDFS-11993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: chencan
>Assignee: chencan
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-11993.002.patch, HADOOP-11993.patch
>
>
> In function BlockSeekTo, when connect faild to datanode socket address,log as
> follow:
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
> + ", add to deadNodes and continue. " + ex, ex);
> add block info may be more explicit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11993) Add log info when connect to datanode socket address failed
[
https://issues.apache.org/jira/browse/HDFS-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-11993:
Fix Version/s: 3.0.0-alpha4
2.9.0
> Add log info when connect to datanode socket address failed
> ---
>
> Key: HDFS-11993
> URL: https://issues.apache.org/jira/browse/HDFS-11993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: chencan
>Assignee: chencan
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-11993.002.patch, HADOOP-11993.patch
>
>
> In function BlockSeekTo, when connect faild to datanode socket address,log as
> follow:
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
> + ", add to deadNodes and continue. " + ex, ex);
> add block info may be more explicit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11993) Add log info when connect to datanode socket address failed
[
https://issues.apache.org/jira/browse/HDFS-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-11993:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk and cherry-picked into branch-2.
> Add log info when connect to datanode socket address failed
> ---
>
> Key: HDFS-11993
> URL: https://issues.apache.org/jira/browse/HDFS-11993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: chencan
>Assignee: chencan
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-11993.002.patch, HADOOP-11993.patch
>
>
> In function BlockSeekTo, when connect faild to datanode socket address,log as
> follow:
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
> + ", add to deadNodes and continue. " + ex, ex);
> add block info may be more explicit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11993) Add log info when connect to datanode socket address failed
[
https://issues.apache.org/jira/browse/HDFS-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063699#comment-16063699
]
Ravi Prakash commented on HDFS-11993:
-
Thanks Chen Liang for the review and suggestion. Thanks chencan for the
contribution and updating the patch. +1. LGTM
Committing to shortly.
> Add log info when connect to datanode socket address failed
> ---
>
> Key: HDFS-11993
> URL: https://issues.apache.org/jira/browse/HDFS-11993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: chencan
>Assignee: chencan
> Attachments: HADOOP-11993.002.patch, HADOOP-11993.patch
>
>
> In function BlockSeekTo, when connect faild to datanode socket address,log as
> follow:
> DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
> + ", add to deadNodes and continue. " + ex, ex);
> add block info may be more explicit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11078) Fix NPE in LazyPersistFileScrubber
[ https://issues.apache.org/jira/browse/HDFS-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11078: --- Summary: Fix NPE in LazyPersistFileScrubber (was: NPE in LazyPersistFileScrubber) > Fix NPE in LazyPersistFileScrubber > -- > > Key: HDFS-11078 > URL: https://issues.apache.org/jira/browse/HDFS-11078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2 > > Attachments: HDFS-11078.000.patch, HDFS-11078.001.patch, > HDFS-11078-branch-2.7.patch > > > If a block is removed, it will be removed from the block map. When the > clearCorruptLazyPersistFiles() tries to delete the block, it may already be > deleted and generate a null pointer exception. > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.clearCorruptLazyPersistFiles(FSNamesystem.java:3820) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:3851) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12042) Reduce memory used by snapshot diff data structures
Misha Dmitriev created HDFS-12042:
-
Summary: Reduce memory used by snapshot diff data structures
Key: HDFS-12042
URL: https://issues.apache.org/jira/browse/HDFS-12042
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Misha Dmitriev
Assignee: Misha Dmitriev
When snapshot diff operation is performed in a NameNode that manages several
million HDFS files/directories, NN needs a lot of memory. Some of that memory
is wasted due to suboptimal data structures, such as empty or under-populated
ArrayLists, etc. Analyzing one heap dump with jxray (www.jxray.com), we
observed the following problems with data structures:
{code}
9. BAD COLLECTIONS
Total collections: 99,707,902 Bad collections: 88,799,760 Overhead:
9,063,898K (18.2%)
Top bad collections:
Ovhd Problem Num objs Type
-
3,056,014K (6.1%) small 29435572 j.u.ArrayList
2,641,373K (5.3%) 1-elem 21837906 j.u.ArrayList
864,215K (1.7%) 1-elem 5291813 j.u.TreeSet
808,456K (1.6%) 1-elem 3045847 j.u.HashMap
602,470K (1.2%) empty 18549109 j.u.ArrayList
441,563K (0.9%) empty 4356975 j.u.TreeSet
373,088K (0.7%) empty 5297007 j.u.HashMap
270,324K (0.5%) small 931394 j.u.HashMap
{code}
The data structures created by HDFS code that suffer from the above problems
are, in particular:
{code}
4,228,182K (8.5%): j.u.ArrayList: 19412263 of small 2,111,087K (4.2%),
12932408 of 1-elem 1,717,585K (3.4%), 12784
310 of empty 399,509K (0.8%)
<-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
org.apache.hadoop.hdfs.server.nameno
de.snapshot.FileWithSnapshotFeature.diffs <--
org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <-- org.apache.
hadoop.hdfs.server.namenode.INodeFile.features <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- or
g.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.e
ntries <-- org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop.hdfs.server.blockman
agement.BlocksMap$1.entries <--
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
org.apache.hadoop
.hdfs.server.blockmanagement.BlockManager.blocksMap <--
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$B
lockReportProcessingThread.this$0 <-- j.l.Thread[] <-- j.l.ThreadGroup.threads
<-- j.l.Thread.group <-- Java Static:
org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
{code}
and
{code}
575,557K (1.2%): j.u.ArrayList: 4363271 of 1-elem 409,056K (0.8%), 2439001 of
small 166,482K (0.3%)
<-- org.apache.hadoop.hdfs.server.namenode.INodeDirectory.children <--
org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
org.apache.hadoop.util.LightWeightGSet.entries <--
org.apache.hadoop.hdfs.server.namenode.INodeMap.map <--
org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
<-- org.apache.hadoop.util.Daemon.target <--
org.apache.hadoop.hdfs.server.namenode.FSDirectory.inodeMap <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.dir <--
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.this$0
<-- org.apache.hadoop.util.Daemon.target <-- j.l.Thread[] <--
j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java Static:
org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
{code}
There are several different reference chains that all lead to
FileDiffList.diffs or INodeDirectory.children. The total percentage of memory
wasted by these data structures in the analyzed dump is about 12%. By creating
these lists lazily and/or with capacity that better matches their actual size,
we should be able to reclaim a significant part of these 12%.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[
https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063669#comment-16063669
]
Hadoop QA commented on HDFS-12018:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874534/HDFS-12018-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 969b722c5839 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 31eafb1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20048/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20048/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
> ---
>
>
[jira] [Commented] (HDFS-12008) Improve the available-space block placement policy
[
https://issues.apache.org/jira/browse/HDFS-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063658#comment-16063658
]
Kihwal Lee commented on HDFS-12008:
---
Test failures are not caused by the patch. {{TestFsDatasetImpl}} failed when I
ran without the patch. HDFS-12040.
{{TestDataNodeVolumeFailureReporting}} seems flaky and times out occasionally.
HDFS-11488.
All other tests pass consistently when run locally.
> Improve the available-space block placement policy
> --
>
> Key: HDFS-12008
> URL: https://issues.apache.org/jira/browse/HDFS-12008
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: block placement
>Affects Versions: 2.8.1
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-12008.patch, HDFS-12008.v2.branch-2.patch,
> HDFS-12008.v2.trunk.patch, HDFS-12008.v2.trunk.patch,
> RandomAllocationPolicy.png
>
>
> AvailableSpaceBlockPlacementPolicy currently picks two nodes unconditionally,
> then picks one node. It could avoid picking the second node when not
> necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12033) DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks.
[
https://issues.apache.org/jira/browse/HDFS-12033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12033:
-
Attachment: HDFS-12033.01.patch
Thanks for the suggestions, [~andrew.wang]
Added a new test in 01 patch.
> DatanodeManager picking EC recovery tasks should also consider the number of
> regular replication tasks.
> ---
>
> Key: HDFS-12033
> URL: https://issues.apache.org/jira/browse/HDFS-12033
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12033.00.patch, HDFS-12033.01.patch
>
>
> In {{DatanodeManager#handleHeartbeat}}, it choose both pending replication
> list and pending EC list to up to {{maxTransfers}} items.
> It should only send {{maxTransfers}} tasks combined to DN.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063639#comment-16063639 ] Xiaoyu Yao commented on HDFS-12018: --- Patch v3 looks good to me. +1 pending Jenkins. > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch, HDFS-12018-HDFS-7240.003.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063608#comment-16063608
]
Hadoop QA commented on HDFS-9079:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-9079 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-9079 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12775130/HDFS-9079.15.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20049/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9079.01.patch, HDFS-9079.02.patch,
> HDFS-9079.03.patch, HDFS-9079.04.patch, HDFS-9079.05.patch,
> HDFS-9079.06.patch, HDFS-9079.07.patch, HDFS-9079.08.patch,
> HDFS-9079.09.patch, HDFS-9079.10.patch, HDFS-9079.11.patch,
> HDFS-9079.12.patch, HDFS-9079.13.patch, HDFS-9079.14.patch,
> HDFS-9079.15.patch, HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> With multiple streamer threads run in parallel, we need to correctly handle a
> large number of possible combinations of interleaved thread events. For
> example, {{streamer_B}} starts step 2 in between events {{streamer_A.2}} and
> {{streamer_A.3}}.
> HDFS-9040 moves steps 1, 2, 3, 6 from streamer to {{DFSStripedOutputStream}}.
> This JIRA proposes some further optimizations based on HDFS-9040:
> # We can preallocate GS when NN creates a new striped block group
> ({{FSN#createNewBlock}}). For each new striped block group we can reserve
> {{NUM_PARITY_BLOCKS}} GS's. If more than {{NUM_PARITY_BLOCKS}} errors have
> happened we shouldn't try to further recover anyway.
> # We can use a dedicated event processor to offload the error handling logic
> from {{DFSStripedOutputStream}}, which is not a long running daemon.
> # We can limit the lifespan of a streamer to be a single block. A streamer
> ends either after finishing the current block or when encountering a DN
> failure.
> With the proposed change, a {{StripedDataStreamer}}'s flow becomes:
> {code}
> 1) Finds DN error => 2) Notify coordinator (async, not waiting for response)
> => terminates
> 1) Finds external error => 2) Applies new GS to DN (createBlockOutputStream)
> => 3) Ack from DN => 4) Notify coordinator (async, not waiting for response)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9079:
--
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9079.01.patch, HDFS-9079.02.patch,
> HDFS-9079.03.patch, HDFS-9079.04.patch, HDFS-9079.05.patch,
> HDFS-9079.06.patch, HDFS-9079.07.patch, HDFS-9079.08.patch,
> HDFS-9079.09.patch, HDFS-9079.10.patch, HDFS-9079.11.patch,
> HDFS-9079.12.patch, HDFS-9079.13.patch, HDFS-9079.14.patch,
> HDFS-9079.15.patch, HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> With multiple streamer threads run in parallel, we need to correctly handle a
> large number of possible combinations of interleaved thread events. For
> example, {{streamer_B}} starts step 2 in between events {{streamer_A.2}} and
> {{streamer_A.3}}.
> HDFS-9040 moves steps 1, 2, 3, 6 from streamer to {{DFSStripedOutputStream}}.
> This JIRA proposes some further optimizations based on HDFS-9040:
> # We can preallocate GS when NN creates a new striped block group
> ({{FSN#createNewBlock}}). For each new striped block group we can reserve
> {{NUM_PARITY_BLOCKS}} GS's. If more than {{NUM_PARITY_BLOCKS}} errors have
> happened we shouldn't try to further recover anyway.
> # We can use a dedicated event processor to offload the error handling logic
> from {{DFSStripedOutputStream}}, which is not a long running daemon.
> # We can limit the lifespan of a streamer to be a single block. A streamer
> ends either after finishing the current block or when encountering a DN
> failure.
> With the proposed change, a {{StripedDataStreamer}}'s flow becomes:
> {code}
> 1) Finds DN error => 2) Notify coordinator (async, not waiting for response)
> => terminates
> 1) Finds external error => 2) Applies new GS to DN (createBlockOutputStream)
> => 3) Ack from DN => 4) Notify coordinator (async, not waiting for response)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11956) Do not require a storage ID or target storage IDs when writing a block
[
https://issues.apache.org/jira/browse/HDFS-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063595#comment-16063595
]
Hudson commented on HDFS-11956:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11925 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11925/])
HDFS-11956. Do not require a storage ID or target storage IDs when (wang: rev
2c367b464c86a7d67a2b8dd82ae804d169957573)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/token/block/TestBlockToken.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
> Do not require a storage ID or target storage IDs when writing a block
> --
>
> Key: HDFS-11956
> URL: https://issues.apache.org/jira/browse/HDFS-11956
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Ewan Higgs
>Priority: Blocker
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11956.001.patch, HDFS-11956.002.patch,
> HDFS-11956.003.patch, HDFS-11956.004.patch
>
>
> Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
> When talking to a 3.0.0-alpha4 DN with security on:
> {noformat}
> 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block token verification failed: op=WRITE_BLOCK,
> remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
> [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with
> StorageIDs []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[
https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063592#comment-16063592
]
Zhe Zhang commented on HDFS-11882:
--
Sorry I missed this ping earlier.
bq. Although it says that "at least data block number size cells", the method
doesn't check this.
The current logic in {{getNumAckedStripes}} actually seems to get the number of
acked full stripes. With 6+3 scheme, {{numAllBlocks}} is 9, and the logic takes
the minimum number of acked stripes among all 9 streamers.
So I don't really understand why {{ackedBytes}} could be greater than
{{sentBytes}}. Agreed with [~andrew.wang] that a unit test reproducing the
behavior would be ideal.
Also, I don't have time to continue working on HDFS-9079 any more but I feel it
is needed for the complex EC writer logic. [~andrew.wang] [~drankye] Would be
great if you can take a look and see if it's worth pursuing.
> Client fails if acknowledged size is greater than bytes sent
>
>
> Key: HDFS-11882
> URL: https://issues.apache.org/jira/browse/HDFS-11882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11882.01.patch
>
>
> Some tests of erasure coding fails by the following exception. The following
> test was removed by HDFS-11823, however, this type of error can happen in
> real cluster.
> {noformat}
> Running
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure)
> Time elapsed: 38.831 sec <<< ERROR!
> java.lang.IllegalStateException: null
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-11992:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
Committed this to branch-2. Thanks [~xiaodong.hu]!
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch,
> HDFS-11992-branch-2.001.patch, HDFS-11992.branch-2.01.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12018: -- Attachment: HDFS-12018-HDFS-7240.003.patch Post v003 patch to rebase. > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch, HDFS-12018-HDFS-7240.003.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11956) Do not require a storage ID or target storage IDs when writing a block
[
https://issues.apache.org/jira/browse/HDFS-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11956:
---
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
Status: Resolved (was: Patch Available)
Thanks for working on this Ewan, committed to trunk! Thanks also to Wei-chiu
for reviewing.
> Do not require a storage ID or target storage IDs when writing a block
> --
>
> Key: HDFS-11956
> URL: https://issues.apache.org/jira/browse/HDFS-11956
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Ewan Higgs
>Priority: Blocker
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11956.001.patch, HDFS-11956.002.patch,
> HDFS-11956.003.patch, HDFS-11956.004.patch
>
>
> Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
> When talking to a 3.0.0-alpha4 DN with security on:
> {noformat}
> 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block token verification failed: op=WRITE_BLOCK,
> remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
> [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with
> StorageIDs []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[
https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063551#comment-16063551
]
Hadoop QA commented on HDFS-12018:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-12018 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874530/HDFS-12018-HDFS-7240.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20047/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
> ---
>
> Key: HDFS-12018
> URL: https://issues.apache.org/jira/browse/HDFS-12018
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Chen Liang
>Priority: Trivial
> Attachments: HDFS-12018-HDFS-7240.001.patch,
> HDFS-12018-HDFS-7240.002.patch
>
>
> We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the
> issue that we need to the same for CBlocks, since CBlock uses Ozone's Config
> files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12032) Inaccurate comment on DatanodeDescriptor#getNumberOfBlocksToBeErasureCoded
[
https://issues.apache.org/jira/browse/HDFS-12032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063550#comment-16063550
]
Hudson commented on HDFS-12032:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11924 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11924/])
HDFS-12032. Inaccurate comment on (wang: rev
06c8ca3bb330c1763d31ed37309e7552dcd3e7aa)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> Inaccurate comment on DatanodeDescriptor#getNumberOfBlocksToBeErasureCoded
> --
>
> Key: HDFS-12032
> URL: https://issues.apache.org/jira/browse/HDFS-12032
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Trivial
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-12032.001.patch
>
>
> I saw this comment is an inaccurate copy paste:
> {noformat}
> /**
>* The number of work items that are pending to be replicated
>*/
> @VisibleForTesting
> public int getNumberOfBlocksToBeErasureCoded() {
> return erasurecodeBlocks.size();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11956) Do not require a storage ID or target storage IDs when writing a block
[
https://issues.apache.org/jira/browse/HDFS-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11956:
---
Summary: Do not require a storage ID or target storage IDs when writing a
block (was: Fix BlockToken compatibility with Hadoop 2.x clients)
> Do not require a storage ID or target storage IDs when writing a block
> --
>
> Key: HDFS-11956
> URL: https://issues.apache.org/jira/browse/HDFS-11956
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HDFS-11956.001.patch, HDFS-11956.002.patch,
> HDFS-11956.003.patch, HDFS-11956.004.patch
>
>
> Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
> When talking to a 3.0.0-alpha4 DN with security on:
> {noformat}
> 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block token verification failed: op=WRITE_BLOCK,
> remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
> [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with
> StorageIDs []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11956) Fix BlockToken compatibility with Hadoop 2.x clients
[
https://issues.apache.org/jira/browse/HDFS-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11956:
---
Release Note: Hadoop 2.x clients do not pass the storage ID or target
storage IDs when writing a block. For backwards compatibility, the DataNode
will not require the presence of these fields. This means older clients are
unable to write to a particular storage as chosen by the NameNode (e.g.
HDFS-9806). (was: Introduce dfs.block.access.token.storageid.enable which will
be false by default. When it's turned on, the
BlockTokenSecretManager.checkAccess will consider the storage ID when verifying
the request. This allows for backwards compatibility all the way back to 2.6.x.)
> Fix BlockToken compatibility with Hadoop 2.x clients
>
>
> Key: HDFS-11956
> URL: https://issues.apache.org/jira/browse/HDFS-11956
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HDFS-11956.001.patch, HDFS-11956.002.patch,
> HDFS-11956.003.patch, HDFS-11956.004.patch
>
>
> Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
> When talking to a 3.0.0-alpha4 DN with security on:
> {noformat}
> 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block token verification failed: op=WRITE_BLOCK,
> remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
> [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with
> StorageIDs []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12018: -- Attachment: HDFS-12018-HDFS-7240.002.patch Thanks [~xyao] for the comments! Post v002 patch. The last two changes you suggested requires additional change to a couple places in cblock code, I have filed HDFS-12041 to follow up. The other comments are addressed. > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch, > HDFS-12018-HDFS-7240.002.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11956) Fix BlockToken compatibility with Hadoop 2.x clients
[
https://issues.apache.org/jira/browse/HDFS-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063533#comment-16063533
]
Andrew Wang commented on HDFS-11956:
+1 LGTM, I'll adjust the release notes as well.
> Fix BlockToken compatibility with Hadoop 2.x clients
>
>
> Key: HDFS-11956
> URL: https://issues.apache.org/jira/browse/HDFS-11956
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Andrew Wang
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HDFS-11956.001.patch, HDFS-11956.002.patch,
> HDFS-11956.003.patch, HDFS-11956.004.patch
>
>
> Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
> When talking to a 3.0.0-alpha4 DN with security on:
> {noformat}
> 2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block token verification failed: op=WRITE_BLOCK,
> remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
> [DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with
> StorageIDs []
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-12040:
--
Summary: TestFsDatasetImpl.testCleanShutdownOfVolume fails (was:
TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2)
> TestFsDatasetImpl.testCleanShutdownOfVolume fails
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063528#comment-16063528
]
Kihwal Lee commented on HDFS-12040:
---
I just ran it on trunk and it failed.
{noformat}
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.152 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
Time elapsed: 5.934 sec <<< ERROR!
java.lang.IllegalArgumentException: Total wait time should be greater than
check interval time
{noformat}
> TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12041) Block Storage : make the server address config more concise
Chen Liang created HDFS-12041:
-
Summary: Block Storage : make the server address config more
concise
Key: HDFS-12041
URL: https://issues.apache.org/jira/browse/HDFS-12041
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Chen Liang
Priority: Minor
Currently there are a few places where the address are read from config like
such
{code}
String cbmIPAddress = ozoneConf.get(
DFS_CBLOCK_JSCSI_CBLOCK_SERVER_ADDRESS_KEY,
DFS_CBLOCK_JSCSI_CBLOCK_SERVER_ADDRESS_DEFAULT
);
int cbmPort = ozoneConf.getInt(
DFS_CBLOCK_JSCSI_PORT_KEY,
DFS_CBLOCK_JSCSI_PORT_DEFAULT
);
{code}
Similarly for jscsi address config. Maybe we should consider merge these to one
single key config in form of host:port.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12032) Inaccurate comment on DatanodeDescriptor#getNumberOfBlocksToBeErasureCoded
[
https://issues.apache.org/jira/browse/HDFS-12032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-12032:
---
Resolution: Fixed
Fix Version/s: 3.0.0-alpha4
Status: Resolved (was: Patch Available)
Thanks Eddy for reviewing! Committed to trunk.
> Inaccurate comment on DatanodeDescriptor#getNumberOfBlocksToBeErasureCoded
> --
>
> Key: HDFS-12032
> URL: https://issues.apache.org/jira/browse/HDFS-12032
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Trivial
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-12032.001.patch
>
>
> I saw this comment is an inaccurate copy paste:
> {noformat}
> /**
>* The number of work items that are pending to be replicated
>*/
> @VisibleForTesting
> public int getNumberOfBlocksToBeErasureCoded() {
> return erasurecodeBlocks.size();
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12030) Ozone: CLI: support infoKey command
[
https://issues.apache.org/jira/browse/HDFS-12030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063503#comment-16063503
]
Xiaoyu Yao commented on HDFS-12030:
---
Thanks [~linyiqun] for working on this. The patch looks good to me overall.
Here are a few comments:
*KeyHandler.java*
Line 55: remove the unused parameter info
Line 63-108: we should have a separate procesor for getKeyInfo as commented in
the Keys.java below.
*Keys.java*
Line 90-99: Since this is purely a KSM metadata operation, I think we should
define a new operation
like getKeyInfo for Keys interface instead of mixing with the existing one to
get the key designed for streaming data.
*DistributedStorageHandler.java*
Line 441/448: I don't think we should set the keySize here. Many times, the
callers make this call to figure
out the size of the key from KSM.
*Test*
Can you add a test to the oz shell command infoKey?
> Ozone: CLI: support infoKey command
> ---
>
> Key: HDFS-12030
> URL: https://issues.apache.org/jira/browse/HDFS-12030
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Yiqun Lin
> Attachments: HDFS-12030-HDFS-7240.001.patch
>
>
> {code}
> HW11717:ozone xyao$ hdfs oz -infoKey
> http://localhost:9864/vol-2/bucket-1/key-1 -user xyao
> Command Failed : {"httpCode":0,"shortMessage":"Not supported
> yet","resource":null,"message":"Not supported
> yet","requestID":null,"hostName":null}
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063487#comment-16063487
]
Akira Ajisaka commented on HDFS-11992:
--
LGTM, +1. All the failed tests except TestFsDatasetImpl passed locally.
TestFsDatasetImpl is failing but it is not related to the patch. Filed
HDFS-12040.
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch,
> HDFS-11992-branch-2.001.patch, HDFS-11992.branch-2.01.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2
[
https://issues.apache.org/jira/browse/HDFS-12040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-12040:
-
Description:
Found when reviewing HDFS-11992
{noformat}
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
Time elapsed: 6.055 sec <<< ERROR!
java.lang.IllegalArgumentException: Total wait time should be greater than
check interval time
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
{noformat}
was:
{noformat}
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
Time elapsed: 6.055 sec <<< ERROR!
java.lang.IllegalArgumentException: Total wait time should be greater than
check interval time
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
{noformat}
> TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2
> -
>
> Key: HDFS-12040
> URL: https://issues.apache.org/jira/browse/HDFS-12040
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>
> Found when reviewing HDFS-11992
> {noformat}
> Running
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 6.055 sec <<< ERROR!
> java.lang.IllegalArgumentException: Total wait time should be greater than
> check interval time
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12040) TestFsDatasetImpl.testCleanShutdownOfVolume fails in branch-2
Akira Ajisaka created HDFS-12040:
Summary: TestFsDatasetImpl.testCleanShutdownOfVolume fails in
branch-2
Key: HDFS-12040
URL: https://issues.apache.org/jira/browse/HDFS-12040
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
{noformat}
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.208 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
testCleanShutdownOfVolume(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
Time elapsed: 6.055 sec <<< ERROR!
java.lang.IllegalArgumentException: Total wait time should be greater than
check interval time
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:278)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testCleanShutdownOfVolume(TestFsDatasetImpl.java:670)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12028) Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
[
https://issues.apache.org/jira/browse/HDFS-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063457#comment-16063457
]
Chen Liang commented on HDFS-12028:
---
Thanks [~xyao] for the comments. [~cheersyang] it seems to have resolved in my
environment, could you share a little bit more about how you got this? (e.g.
how and which commands you ran etc). Also, you may need to mvn clean and build
the whole thing again to make the patch take effect.
> Ozone: CLI: remove noisy slf4j binding output from hdfs oz command.
> ---
>
> Key: HDFS-12028
> URL: https://issues.apache.org/jira/browse/HDFS-12028
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-12028-HDFS-7240.001.patch
>
>
> Currently when you run CLI "hdfs oz ...", there always a noisy slf4j biding
> issue log output. This ticket is opened to removed it.
> {code}
> xyao$ hdfs oz
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/xyao/deploy/ozone/hadoop-3.0.0-alpha4-SNAPSHOT/share/hadoop/hdfs/lib/logback-classic-1.0.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12018) Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml
[ https://issues.apache.org/jira/browse/HDFS-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063447#comment-16063447 ] Xiaoyu Yao commented on HDFS-12018: --- Thanks [~vagarychen] for working on this. The patch looks good to me. I just have a few minor issues below: *ozone-default.xml* Line 520: dfs.cblock.cache.core.pool.size -> dfs.cblock.cache.core.min.pool.size to be consistent with the dfs.cblock.cache.max.pool.size Line 523: Minimum number of thread pool thread that cBlock cache will use for background I/O. Line 531: Maximum number of thread pool thread that cBlcok cache will use for background I/O. Line 548: "Can be 1, 5 or 10 for now" -> "Supported values are 1, 5, 10. Use 1 for high priority and 10 for low priority? Line 569: dfs.cblock.jscsi.server.address can this be a host:port or just host or both? Line 578: same as above > Ozone: Misc: Add CBlocks defaults to ozone-defaults.xml > --- > > Key: HDFS-12018 > URL: https://issues.apache.org/jira/browse/HDFS-12018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12018-HDFS-7240.001.patch > > > We have just updated ozone-defaults.xml in HDFS-11990. This JIRA tracks the > issue that we need to the same for CBlocks, since CBlock uses Ozone's Config > files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11589) Unable to remove dead node after datanode decommission
[ https://issues.apache.org/jira/browse/HDFS-11589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11589. --- Resolution: Duplicate > Unable to remove dead node after datanode decommission > --- > > Key: HDFS-11589 > URL: https://issues.apache.org/jira/browse/HDFS-11589 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Chang Zong > > After successfully decommissioned a datanode, the state of that node in web > ui(http://:50070) is changed to "decommissioned". Then I > stopped the datanode service in that host and the state is changed to "Dead", > and the number of last contact keeps increasing. I tried to remove the > hostname from the dfs.exclude and run "hdfs dfsadmin -refreshNodes" again, > but that dead node is still there. > Any way to solve that problem without restarting the whole cluster? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12004) Namenode UI continues to list DNs that have been removed from include and exclude
[ https://issues.apache.org/jira/browse/HDFS-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen resolved HDFS-12004. Resolution: Duplicate > Namenode UI continues to list DNs that have been removed from include and > exclude > - > > Key: HDFS-12004 > URL: https://issues.apache.org/jira/browse/HDFS-12004 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Erik Krogen >Priority: Minor > > Initially in HDFS, after a DN was decommission and subsequently removed from > the exclude file (thus removing all references to it), it would still appear > in the NN UI as a "dead" node until the NN was restarted. In HDFS-1773, > discussion about this was had, and it was decided that the web UI should not > show these nodes. However when HDFS-5334 went through and the NN web UI was > reimplemented client-side, the behavior reverted back to pre-HDFS-1773, and > dead+decommissioned nodes once again showed in the dead list. This can be > operationally confusing for the same reasons as discussed in HDFS-1773. > I would like to open this discussion to determine if the regression was > intentional or if we should carry forward the logic implemented HDFS-1773 > into the new UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12004) Namenode UI continues to list DNs that have been removed from include and exclude
[ https://issues.apache.org/jira/browse/HDFS-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063339#comment-16063339 ] Erik Krogen commented on HDFS-12004: Ah, don't know how I missed that. Thank you, [~kihwal]. Sorry for the duplication. > Namenode UI continues to list DNs that have been removed from include and > exclude > - > > Key: HDFS-12004 > URL: https://issues.apache.org/jira/browse/HDFS-12004 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Erik Krogen >Priority: Minor > > Initially in HDFS, after a DN was decommission and subsequently removed from > the exclude file (thus removing all references to it), it would still appear > in the NN UI as a "dead" node until the NN was restarted. In HDFS-1773, > discussion about this was had, and it was decided that the web UI should not > show these nodes. However when HDFS-5334 went through and the NN web UI was > reimplemented client-side, the behavior reverted back to pre-HDFS-1773, and > dead+decommissioned nodes once again showed in the dead list. This can be > operationally confusing for the same reasons as discussed in HDFS-1773. > I would like to open this discussion to determine if the regression was > intentional or if we should carry forward the logic implemented HDFS-1773 > into the new UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12004) Namenode UI continues to list DNs that have been removed from include and exclude
[ https://issues.apache.org/jira/browse/HDFS-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063279#comment-16063279 ] Kihwal Lee commented on HDFS-12004: --- I thought it was fixed in HDFS-8950. > Namenode UI continues to list DNs that have been removed from include and > exclude > - > > Key: HDFS-12004 > URL: https://issues.apache.org/jira/browse/HDFS-12004 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Erik Krogen >Priority: Minor > > Initially in HDFS, after a DN was decommission and subsequently removed from > the exclude file (thus removing all references to it), it would still appear > in the NN UI as a "dead" node until the NN was restarted. In HDFS-1773, > discussion about this was had, and it was decided that the web UI should not > show these nodes. However when HDFS-5334 went through and the NN web UI was > reimplemented client-side, the behavior reverted back to pre-HDFS-1773, and > dead+decommissioned nodes once again showed in the dead list. This can be > operationally confusing for the same reasons as discussed in HDFS-1773. > I would like to open this discussion to determine if the regression was > intentional or if we should carry forward the logic implemented HDFS-1773 > into the new UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12039) Ozone: Implement update volume owner in ozone shell
Weiwei Yang created HDFS-12039:
--
Summary: Ozone: Implement update volume owner in ozone shell
Key: HDFS-12039
URL: https://issues.apache.org/jira/browse/HDFS-12039
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Weiwei Yang
Ozone shell command {{updateVolume}} should support to update the owner of a
volume, using following syntax
{code}
hdfs oz -updateVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -owner xyz
-root
{code}
this could work from rest api, following command could change the volume owner
to {{www}}
{code}
curl -X PUT -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1"
-H "x-ozone-user:www" -H "Authorization:OZONE root"
http://ozone1.fyre.ibm.com:9864/volume-wwei-0
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12038) Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself
[
https://issues.apache.org/jira/browse/HDFS-12038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063229#comment-16063229
]
Weiwei Yang commented on HDFS-12038:
Seems like there is a bug in {{InfoVolumeHandler}}
line 82 - 86
{code}
if (rootName != null) {
client.setUserAuth(rootName);
} else {
client.setUserAuth(userName);
}
{code}
line 89
{code}
client.setUserAuth(rootName);
{code}
if rootName is null, this code will null to user auth, which probably causes
the error in the description.
> Ozone: Non-admin user is unable to run InfoVolume to the volume owned by
> itself
> ---
>
> Key: HDFS-12038
> URL: https://issues.apache.org/jira/browse/HDFS-12038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>
> Reproduce steps
> 1. Create a volume with a non-admin user
> {code}
> hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user
> wwei -root -quota 2TB
> {code}
> 2. Run infoVolume command to get this volume info
> {noformat}
> hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
> Command Failed :
> {"httpCode":400,"shortMessage":"badAuthorization","resource":null,"message":"Missing
> authorization or authorization has to be
> unique.","requestID":"221efb47-72b9-498d-ac19-907257428573","hostName":"ozone1.fyre.ibm.com"}
> {noformat}
> add {{-root}} to run as admin user could bypass this issue
> {noformat}
> hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
> -root
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 2
> },
> "volumeName" : "volume-wwei-0",
> "createdOn" : null,
> "createdBy" : "hdfs"
> }
> {noformat}
> expecting: both volume owner and admin should be able to run infoVolume
> command.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12038) Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself
Weiwei Yang created HDFS-12038:
--
Summary: Ozone: Non-admin user is unable to run InfoVolume to the
volume owned by itself
Key: HDFS-12038
URL: https://issues.apache.org/jira/browse/HDFS-12038
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Weiwei Yang
Reproduce steps
1. Create a volume with a non-admin user
{code}
hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
-root -quota 2TB
{code}
2. Run infoVolume command to get this volume info
{noformat}
hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
Command Failed :
{"httpCode":400,"shortMessage":"badAuthorization","resource":null,"message":"Missing
authorization or authorization has to be
unique.","requestID":"221efb47-72b9-498d-ac19-907257428573","hostName":"ozone1.fyre.ibm.com"}
{noformat}
add {{-root}} to run as admin user could bypass this issue
{noformat}
hdfs oz -infoVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user wwei
-root
{
"owner" : {
"name" : "wwei"
},
"quota" : {
"unit" : "TB",
"size" : 2
},
"volumeName" : "volume-wwei-0",
"createdOn" : null,
"createdBy" : "hdfs"
}
{noformat}
expecting: both volume owner and admin should be able to run infoVolume command.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12037) Ozone: Improvement rest API output format for better looking
Weiwei Yang created HDFS-12037:
--
Summary: Ozone: Improvement rest API output format for better
looking
Key: HDFS-12037
URL: https://issues.apache.org/jira/browse/HDFS-12037
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Right now ozone rest api output is displayed as a raw json string in single
line, not quite human readable,
{noformat}
{"volumes":[{"owner":{"name":"wwei"},"quota":{"unit":"GB","size":200},"volumeName":"volume-aug-1","createdOn":null,"createdBy":null}]}
{noformat}
propose to improve the output format by pretty printer
{noformat}
{
"volumes" : [ {
"owner" : {
"name" : "wwei"
},
"quota" : {
"unit" : "GB",
"size" : 200
},
"volumeName" : "volume-aug-1",
"createdOn" : null,
"createdBy" : null
} ]
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12008) Improve the available-space block placement policy
[
https://issues.apache.org/jira/browse/HDFS-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063189#comment-16063189
]
Hadoop QA commented on HDFS-12008:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 28 unchanged - 2 fixed = 30 total (was 30) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileCorruption |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12008 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874495/HDFS-12008.v2.trunk.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux af8913a8dcca 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 48f4a22 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20045/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20045/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20045/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve the available-space block placement policy
> --
>
> Key: HDFS-12008
> URL:
[jira] [Commented] (HDFS-12011) Add a new load balancing volume choosing policy
[
https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063115#comment-16063115
]
Hadoop QA commented on HDFS-12011:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 498 unchanged - 0 fixed = 501 total (was 498) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874484/HADOOP-12011.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 67c79e1ac11a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 379f19a |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20044/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/20044/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20044/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20044/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20044/console |
| Powered by | Apache
[jira] [Updated] (HDFS-11773) Ozone: KSM : add listVolumes
[ https://issues.apache.org/jira/browse/HDFS-11773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11773: --- Attachment: listVolume_tests.log > Ozone: KSM : add listVolumes > > > Key: HDFS-11773 > URL: https://issues.apache.org/jira/browse/HDFS-11773 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11773-HDFS-7240.001.patch, listVolume_tests.log > > > List volume call can be used in three different contexts. One is for the > administrators to list all volumes in a cluster. Second is for the > administrator to list the volumes owned by a specific user. Third is a user > listing the volumes owned by himself/herself. > Since these calls can return large number of entries the rest protocol > supports paging. Paging is supported by the use of prevKey, prefix and > maxKeys. The caller is aware the this call is neither atomic nor consistent. > So we can iterate over the list even while changes are happening to the list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
