[jira] [Commented] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558205#comment-15558205
]
Hadoop QA commented on HDFS-10974:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 31s{color} | {color:orange} root: The patch generated 2 new + 142 unchanged
- 1 fixed = 144 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 56s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
25s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cli.TestCLI |
| | hadoop.ipc.TestIPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10974 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832263/HDFS-10974.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1212131e0cc2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d10621 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Commented] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[ https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558170#comment-15558170 ] Brahma Reddy Battula commented on HDFS-10987: - Test failures are unrelated..Kindly review. > Make Decommission less expensive when lot of blocks present. > > > Key: HDFS-10987 > URL: https://issues.apache.org/jira/browse/HDFS-10987 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-10987.patch > > > When user want to decommission a node which having 50M blocks +,it could hold > the namesystem lock for long time.We've seen it is taking 36 sec+. > As we knew during this time, Namenode will not available... As this > decommission will continuosly run till all the blocks got replicated,hence > Namenode will unavailable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8555) Random read support on HDFS files using Indexed Namenode feature
[ https://issues.apache.org/jira/browse/HDFS-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559026#comment-15559026 ] Kai Zheng commented on HDFS-8555: - What's the benefit to build and maintain the indexes inside NameNode in HDFS layer instead of on top of it? You mentioned SQL over HDFS, but currently SQL on Hadoop is mainly supported by all kinds of frameworks (for example, Hive, Spark, Impala) with various data formats (with indexes) such as parquet, orc and carbondata, all being built on HDFS. > Random read support on HDFS files using Indexed Namenode feature > > > Key: HDFS-8555 > URL: https://issues.apache.org/jira/browse/HDFS-8555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Affects Versions: 2.5.2 > Environment: Linux >Reporter: amit sehgal >Assignee: Afzal Saan > Original Estimate: 720h > Remaining Estimate: 720h > > Currently Namenode does not provide support to do random reads. With so many > tools built on top of HDFS solving the use case of Exploratory BI and > providing SQL over HDFS. The need of hour is to reduce the number of blocks > read for a Random read. > E.g. extracting say 10 lines worth of information out of 100GB files should > be reading only those block which can potentially have those 10 lines. > This can be achieved by adding a tagging feature per block in name node, each > block written to HDFS will have tags associated to it stored in index. > Namednode when access via the Indexing feature will use this index native to > reduce the no. of block returned to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Attachment: HDFS-10984.v1.patch Added unit tests for topMetrics changes. Additionally a timeout on JMXGet(). > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch, HDFS-10984.v1.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Status: Patch Available (was: Open) > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch, HDFS-10984.v1.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10986) DFSAdmin should log detailed error message if any
[
https://issues.apache.org/jira/browse/HDFS-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559207#comment-15559207
]
Brahma Reddy Battula commented on HDFS-10986:
-
[~liuml07] thanks for pinging me.. +1 for doing this..
Minor nits:
1) Following import might not require...?
{{import java.net.ConnectException;}}
2) it will be nice,if we've testcase for this..Mainly when port is reachable..
> DFSAdmin should log detailed error message if any
> -
>
> Key: HDFS-10986
> URL: https://issues.apache.org/jira/browse/HDFS-10986
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10986.000.patch
>
>
> There are some subcommands in {{DFSAdmin}} that swallow IOException and give
> very limited error message, if any, to the stderr.
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:9866
> Datanode unreachable.
> $ hdfs dfsadmin -getDatanodeInfo localhost:9866
> Datanode unreachable.
> $ hdfs dfsadmin -evictWriters 127.0.0.1:9866
> $ echo $?
> -1
> {code}
> User is not able to get the exception stack even the LOG level is DEBUG. This
> is not very user friendly. Fortunately, if the port number is not accessible
> (say ), users can infer the detailed error message by IPC logs:
> {code}
> $ hdfs dfsadmin -getBalancerBandwidth 127.0.0.1:
> 2016-10-07 18:01:35,115 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2016-10-07 18:01:36,335 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> .
> 2016-10-07 18:01:45,361 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9690. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 2016-10-07 18:01:45,362 WARN ipc.Client: Failed to connect to server:
> localhost/127.0.0.1:9690: retries get failed due to exceeded maximum allowed
> retries number: 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ...
> at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:2073)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2225)
> Datanode unreachable.
> {code}
> We should fix this by providing detailed error message. Actually, the
> {{DFSAdmin#run}} already handles exception carefully, including:
> # set the exit ret value to -1
> # print the error message
> # log the exception stack trace (in DEBUG level)
> All we need to do is to not swallow exceptions without good reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10988) Improve TestBalancerBandwidth
[ https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10988: Status: Patch Available (was: Open) > Improve TestBalancerBandwidth > - > > Key: HDFS-10988 > URL: https://issues.apache.org/jira/browse/HDFS-10988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Brahma Reddy Battula > Attachments: HDFS-10988.patch > > > This jira will deal the following. > 1) Remove Fixed sleep > 2) Remove unused dnproxy > 3) use try with resources -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559313#comment-15559313
]
Kai Zheng commented on HDFS-10974:
--
bq. The replication factor of an EC file is 0 from a user perspective. This
affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs
-stat, webhdfs, httpfs...etc.
I'm a little confused. Why is it {{0}} as the replication factor for striped
files? As discussed in HDFS-10971, the better option should be {{1}} instead.
Could we refine the value?
> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10974.001.patch
>
>
> The replication factor of an EC file is 0 from a user perspective. This
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is
> the command {{hdfs dfs -setrep}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10988) Refactor TestBalancerBandwidth
[ https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10988: Summary: Refactor TestBalancerBandwidth (was: Improve TestBalancerBandwidth) > Refactor TestBalancerBandwidth > -- > > Key: HDFS-10988 > URL: https://issues.apache.org/jira/browse/HDFS-10988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Brahma Reddy Battula > Attachments: HDFS-10988.patch > > > This jira will deal the following. > 1) Remove Fixed sleep > 2) Remove unused dnproxy > 3) use try with resources -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec
[
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559102#comment-15559102
]
Hadoop QA commented on HDFS-10895:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10895 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832309/HDFS-10895-v2.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux ffd513e4ed72 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d10621 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17071/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update HDFS Erasure Coding document about how to enable ISA-L library for
> erasure code codec
>
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS
> codec. With ISA-L, the performance of erasure code RS encode/decode will be
> significantly improved. This task is to add the deploy process into the HDFS
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled
> erasure code policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559194#comment-15559194
]
Hadoop QA commented on HDFS-10984:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 206 unchanged - 0 fixed = 213 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestJMXGet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10984 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832308/HDFS-10984.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 493a4915644d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d10621 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17070/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17070/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Expose nntop output as metrics
> -
>
> Key: HDFS-10984
> URL: https://issues.apache.org/jira/browse/HDFS-10984
>
[jira] [Commented] (HDFS-10988) Refactor TestBalancerBandwidth
[
https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559353#comment-15559353
]
Hadoop QA commented on HDFS-10988:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 9 unchanged - 5 fixed = 11 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.TestCrcCorruption |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832311/HDFS-10988.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dcae37fe7d38 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d10621 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17072/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17072/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17072/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17072/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor TestBalancerBandwidth
> --
>
[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec
[ https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-10895: - Attachment: HDFS-10895-v2.patch Rebase patch against latest trunk doc > Update HDFS Erasure Coding document about how to enable ISA-L library for > erasure code codec > > > Key: HDFS-10895 > URL: https://issues.apache.org/jira/browse/HDFS-10895 > Project: Hadoop HDFS > Issue Type: Task >Reporter: SammiChen >Assignee: SammiChen > Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch > > > Recently, native ISA-L library support is added into HDFS erasure code RS > codec. With ISA-L, the performance of erasure code RS encode/decode will be > significantly improved. This task is to add the deploy process into the HDFS > Erasure Code Document, make it easy for HDFS user to use ISA-L enabled > erasure code policy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10988) Improve TestBalancerBandwidth
Brahma Reddy Battula created HDFS-10988: --- Summary: Improve TestBalancerBandwidth Key: HDFS-10988 URL: https://issues.apache.org/jira/browse/HDFS-10988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brahma Reddy Battula This jira will deal the following. 1) Remove Fixed sleep 2) Remove unused dnproxy 3) use try with resources -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10988) Improve TestBalancerBandwidth
[ https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10988: Attachment: HDFS-10988.patch Uploaded the patch..Kindly Review. > Improve TestBalancerBandwidth > - > > Key: HDFS-10988 > URL: https://issues.apache.org/jira/browse/HDFS-10988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Brahma Reddy Battula > Attachments: HDFS-10988.patch > > > This jira will deal the following. > 1) Remove Fixed sleep > 2) Remove unused dnproxy > 3) use try with resources -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10988) Refactor TestBalancerBandwidth
[ https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-10988: --- Assignee: Brahma Reddy Battula > Refactor TestBalancerBandwidth > -- > > Key: HDFS-10988 > URL: https://issues.apache.org/jira/browse/HDFS-10988 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-10988.patch > > > This jira will deal the following. > 1) Remove Fixed sleep > 2) Remove unused dnproxy > 3) use try with resources -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10984) Expose nntop output as metrics
[
https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557329#comment-15557329
]
Hadoop QA commented on HDFS-10984:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.tools.TestJMXGet |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10984 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832241/HDFS-10984.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 747e7c60cda0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6a38d11 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17067/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17067/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Expose nntop output as metrics
>
[jira] [Updated] (HDFS-10984) Expose nntop output as metrics
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDFS-10984: --- Status: Open (was: Patch Available) Unit test changes pending. > Expose nntop output as metrics > - > > Key: HDFS-10984 > URL: https://issues.apache.org/jira/browse/HDFS-10984 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle > Fix For: 2.7.3 > > Attachments: HDFS-10984.patch > > > The nntop output is already exposed via JMX with HDFS-6982. > However external metrics systems do not get this data. It would be valuable > to track this as a timeseries as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10968) BlockManager#isInNewRack should consider decommissioning nodes
[
https://issues.apache.org/jira/browse/HDFS-10968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557318#comment-15557318
]
Hudson commented on HDFS-10968:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10573 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10573/])
HDFS-10968. BlockManager#isInNewRack should consider decommissioning (jing9:
rev 4d106213c0f4835b723c9a50bd8080a9017122d7)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java
> BlockManager#isInNewRack should consider decommissioning nodes
> --
>
> Key: HDFS-10968
> URL: https://issues.apache.org/jira/browse/HDFS-10968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10968.000.patch
>
>
> For an EC block, it is possible we have enough internal blocks but without
> enough racks. The current reconstruction code calls
> {{BlockManager#isInNewRack}} to check if the target node can increase the
> total rack number for the case, which compares the target node's rack with
> source node racks:
> {code}
> for (DatanodeDescriptor src : srcs) {
> if (src.getNetworkLocation().equals(target.getNetworkLocation())) {
> return false;
> }
> }
> {code}
> However here the {{srcs}} may include a decommissioning node, in which case
> we should allow the target node to be in the same rack with it.
> For e.g., suppose we have 11 nodes: h1 ~ h11, which are located in racks r1,
> r1, r2, r2, r3, r3, r4, r4, r5, r5, r6, respectively. In case that an EC
> block has 9 live internal blocks on (h1~h8 + h11), and one internal block on
> h9 which is to be decommissioned. The current code will not choose h10 for
> reconstruction because isInNewRack thinks h10 is on the same rack with h9.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
Brahma Reddy Battula created HDFS-10987: --- Summary: Make Decommission less expensive when lot of blocks present. Key: HDFS-10987 URL: https://issues.apache.org/jira/browse/HDFS-10987 Project: Hadoop HDFS Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula When user want to decommission a node which having 50M blocks +,it could hold the namesystem lock for long time.We've seen it is taking 36 sec+. As we knew during this time, Namenode will not available... As this decommission will continuosly run till all the blocks got replicated,hence Namenode will unavailable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[ https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10987: Attachment: HDFS-10987.patch Yielding is required in case of single Node have huge number of blocks to process.Introduced same and uploading the patch. > Make Decommission less expensive when lot of blocks present. > > > Key: HDFS-10987 > URL: https://issues.apache.org/jira/browse/HDFS-10987 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-10987.patch > > > When user want to decommission a node which having 50M blocks +,it could hold > the namesystem lock for long time.We've seen it is taking 36 sec+. > As we knew during this time, Namenode will not available... As this > decommission will continuosly run till all the blocks got replicated,hence > Namenode will unavailable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[ https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10987: Status: Patch Available (was: Open) > Make Decommission less expensive when lot of blocks present. > > > Key: HDFS-10987 > URL: https://issues.apache.org/jira/browse/HDFS-10987 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-10987.patch > > > When user want to decommission a node which having 50M blocks +,it could hold > the namesystem lock for long time.We've seen it is taking 36 sec+. > As we knew during this time, Namenode will not available... As this > decommission will continuosly run till all the blocks got replicated,hence > Namenode will unavailable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[
https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557679#comment-15557679
]
Brahma Reddy Battula commented on HDFS-10987:
-
*We can notice from the following, write lock time ~35sec+ . This I seen in
2.7, same issue exists in trunk also.*
*{color:red}2016-09-26 21:09:15,673{color}* | INFO | DecommissionMonitor-0 |
Block: blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
*{color:red}2016-09-26 21:09:45,645{color}* | INFO | DecommissionMonitor-0 |
*{color:blue}Checked 50635805 blocks and 1 nodes this tick{color}* |
DecommissionManager.java:398
*{color:red}2016-09-26 21:09:45,647{color}* | INFO | DecommissionMonitor-0 |
Block: blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
2016-09-26 21:10:15,279 | INFO | DecommissionMonitor-0 | *{color:blue}Checked
50635805 blocks and 1 nodes this tick{color}* | DecommissionManager.java:398
2016-09-26 21:10:15,280 | INFO | DecommissionMonitor-0 | Block:
blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
2016-09-26 21:10:45,595 | INFO | DecommissionMonitor-0 | *{color:blue}Checked
50635805 blocks and 1 nodes this tick{color}* | DecommissionManager.java:398
2016-09-26 21:10:50,346 | INFO | DecommissionMonitor-0 | Block:
blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
2016-09-26 21:11:20,176 | INFO | DecommissionMonitor-0 | *{color:blue}Checked
50635805 blocks and 1 nodes this tick{color}* | DecommissionManager.java:398
2016-09-26 21:11:20,179 | INFO | DecommissionMonitor-0 | Block:
blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
2016-09-26 21:11:49,669 | INFO | DecommissionMonitor-0 | *{color:blue}Checked
50635805 blocks and 1 nodes this tick{color}* | DecommissionManager.java:398
2016-09-26 21:11:49,670 | INFO | DecommissionMonitor-0 | Block:
blk_1075238065_1497241, Expected Replicas: 3, live replicas: 2, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1, excess
replicas: 0, Is Open File: false, Datanodes having this block: X.X.X.X:25009
Y.Y.Y.Y:25009 D.D.D.D:25009 , Current Datanode: D.D.D.D:25009, Is current
datanode decommissioning: true | DecommissionManager.java:299
2016-09-26 21:12:19,192 | INFO | DecommissionMonitor-0 | Checked 50635805
blocks and 1 nodes this tick | DecommissionManager.java:398
> Make Decommission less expensive when lot of blocks present.
>
>
> Key: HDFS-10987
> URL: https://issues.apache.org/jira/browse/HDFS-10987
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> When user want to decommission a node which having 50M blocks +,it could hold
> the namesystem lock for long time.We've seen it is taking 36 sec+.
> As we knew during this time, Namenode will not available... As this
> decommission will continuosly run till all the blocks got replicated,hence
> Namenode will unavailable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10987) Make Decommission less expensive when lot of blocks present.
[
https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557809#comment-15557809
]
Hadoop QA commented on HDFS-10987:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832259/HDFS-10987.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e6e42fda0aff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d10621 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17068/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17068/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Make Decommission less expensive when lot of blocks present.
>
>
> Key: HDFS-10987
> URL: https://issues.apache.org/jira/browse/HDFS-10987
>
[jira] [Commented] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558002#comment-15558002
]
Yiqun Lin commented on HDFS-10974:
--
Thanks for your comments, [~jojochuang]! Attach a initial patch for this.
> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
>
> The replication factor of an EC file is 0 from a user perspective. This
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is
> the command {{hdfs dfs -setrep}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10974:
-
Status: Patch Available (was: Open)
> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
>
> The replication factor of an EC file is 0 from a user perspective. This
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is
> the command {{hdfs dfs -setrep}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10974) Document replication factor for EC files.
[
https://issues.apache.org/jira/browse/HDFS-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10974:
-
Attachment: HDFS-10974.001.patch
> Document replication factor for EC files.
> -
>
> Key: HDFS-10974
> URL: https://issues.apache.org/jira/browse/HDFS-10974
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10974.001.patch
>
>
> The replication factor of an EC file is 0 from a user perspective. This
> affects all user visible commands and APIs, such as {{hdfs dfs -ls}}, {{hdfs
> dfs -stat}}, webhdfs, httpfs...etc.
> In addition, FileSystem#setReplication API for an EC file is ignored. So is
> the command {{hdfs dfs -setrep}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
