[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread Kai Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HDFS-10895:
-
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks [~Sammi] for the contribution!

> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch, HDFS-10895-v5.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread Kai Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HDFS-10895:
-
Fix Version/s: 3.0.0-alpha2

> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch, HDFS-10895-v5.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread Kai Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561379#comment-15561379
 ] 

Kai Zheng commented on HDFS-10895:
--

The latest patch looks good to me. +1

> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch, HDFS-10895-v5.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10924) Add a new instrumented read-write lock

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561359#comment-15561359
 ] 

Hadoop QA commented on HDFS-10924:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10924 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832408/HDFS-10924-4.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 985da11e6788 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / bea004e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17079/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17079/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add a new instrumented read-write lock
> --
>
> Key: HDFS-10924
> URL: https://issues.apache.org/jira/browse/HDFS-10924
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-10924-2.patch, HDFS-10924-3.patch, 
> HDFS-10924-4.patch, HDFS-10924.patch
>
>
> Add a new instrumented read-write lock in hadoop common, so that the 
> HDFS-9668 can use this to improve the locking in FsDatasetImpl



--
This message was sent by

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561349#comment-15561349
 ] 

Hadoop QA commented on HDFS-10972:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 53s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10972 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832404/HDFS-10972.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 69e9f2eff4d5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / bea004e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17078/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17078/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell,

[jira] [Updated] (HDFS-10924) Add a new instrumented read-write lock

2016-10-09 Thread Jingcheng Du (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingcheng Du updated HDFS-10924:

Attachment: HDFS-10924-4.patch

A new patch V4 is uploaded.
Please take a look if it is okay to commit. Thanks a lot.

> Add a new instrumented read-write lock
> --
>
> Key: HDFS-10924
> URL: https://issues.apache.org/jira/browse/HDFS-10924
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-10924-2.patch, HDFS-10924-3.patch, 
> HDFS-10924-4.patch, HDFS-10924.patch
>
>
> Add a new instrumented read-write lock in hadoop common, so that the 
> HDFS-9668 can use this to improve the locking in FsDatasetImpl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10924) Add a new instrumented read-write lock

2016-10-09 Thread Jingcheng Du (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561284#comment-15561284
 ] 

Jingcheng Du edited comment on HDFS-10924 at 10/10/16 5:04 AM:
---

Thanks a lot [~xiaochen] for the comments.
bq. Are the constructors of AutoCloseableReadLock and AutoCloseableWriteLock 
intentionally public? I'm having a hard time coming up with a scenario of this.
Actually no, I have changed them to package private.
bq. Feels like a StringBuilder would be more readable and efficient in 
logWarning, than the current 3 concatenation.
For the log warning, these three strings can be concatenated into one after the 
compiling, this should be fine.
But yes, the StringBuilder is much more efficient than String.format, do we 
need to use StringBuilder instead of String.format, or it's okay to use 
String.format as now?

For others, they are good ones, I will fix them in the next patch V4. Thanks!




was (Author: jingcheng...@intel.com):
Thanks a lot [~xiaochen] for the comments.
bq. Are the constructors of AutoCloseableReadLock and AutoCloseableWriteLock 
intentionally public? I'm having a hard time coming up with a scenario of this.
Actually no, I have changed them to package private.
bq. Feels like a StringBuilder would be more readable and efficient in 
logWarning, than the current 3 concatenation.
For the log warning, these three strings can be concatenated into one after the 
compiling, this should be fine.
But yes, the StringBuilder is much efficient than String.format, do we need to 
use StringBuilder instead of String.format, or it's okay to use String.format 
as now?

For others, they are good ones, I will fix them in the next patch V4. Thanks!



> Add a new instrumented read-write lock
> --
>
> Key: HDFS-10924
> URL: https://issues.apache.org/jira/browse/HDFS-10924
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-10924-2.patch, HDFS-10924-3.patch, 
> HDFS-10924-4.patch, HDFS-10924.patch
>
>
> Add a new instrumented read-write lock in hadoop common, so that the 
> HDFS-9668 can use this to improve the locking in FsDatasetImpl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10982) 'errno' set on successful code path in hdfsOpenFileImpl()

2016-10-09 Thread John Zhuge (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561281#comment-15561281
 ] 

John Zhuge commented on HDFS-10982:
---

The only other {{readDirect}} call in {{hdfsRead}} is fine because it does want 
to pass along {{ENOTSUP}}
{code:title=hdfsRead}
if (f->flags & HDFS_FILE_SUPPORTS_DIRECT_READ) {
  return readDirect(fs, f, buffer, length);
}
{code}

> 'errno' set on successful code path in hdfsOpenFileImpl()
> -
>
> Key: HDFS-10982
> URL: https://issues.apache.org/jira/browse/HDFS-10982
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: libhdfs
>Affects Versions: 2.7.0
>Reporter: Sailesh Mukil
>Assignee: John Zhuge
>  Labels: easyfix
>
> In hdfsOpenFileImpl() in libhdfs/hdfs.c, the following code is used to check 
> if the underlying FileSystem class supports direct reads (i.e. 
> read(ByteBuffer)):
> {code:java}
> if ((flags & O_WRONLY) == 0) {
> // Try a test read to see if we can do direct reads
> char buf;
> if (readDirect(fs, file, , 0) == 0) {
> // Success - 0-byte read should return 0
> file->flags |= HDFS_FILE_SUPPORTS_DIRECT_READ;
> } else if (errno != ENOTSUP) {
> // Unexpected error. Clear it, don't set the direct flag.
> fprintf(stderr,
>   "hdfsOpenFile(%s): WARN: Unexpected error %d when testing "
>   "for direct read compatibility\n", path, errno);
> }
> }
> ret = 0;
> {code}
> The S3A connector, specifically S3AInputStream does not support direct reads, 
> and therefore it sets 'errno = ENOTSUP' on a call to readDirect().
> This 'errno' should be reset before returning the call because this is not an 
> actual error and we're okay with not having direct reads supported.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10924) Add a new instrumented read-write lock

2016-10-09 Thread Jingcheng Du (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561284#comment-15561284
 ] 

Jingcheng Du commented on HDFS-10924:
-

Thanks a lot [~xiaochen] for the comments.
bq. Are the constructors of AutoCloseableReadLock and AutoCloseableWriteLock 
intentionally public? I'm having a hard time coming up with a scenario of this.
Actually no, I have changed them to package private.
bq. Feels like a StringBuilder would be more readable and efficient in 
logWarning, than the current 3 concatenation.
For the log warning, these three strings can be concatenated into one after the 
compiling, this should be fine.
But yes, the StringBuilder is much efficient than String.format, do we need to 
use StringBuilder instead of String.format, or it's okay to use String.format 
as now?

For others, they are good ones, I will fix them in the next patch V4. Thanks!



> Add a new instrumented read-write lock
> --
>
> Key: HDFS-10924
> URL: https://issues.apache.org/jira/browse/HDFS-10924
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-10924-2.patch, HDFS-10924-3.patch, HDFS-10924.patch
>
>
> Add a new instrumented read-write lock in hadoop common, so that the 
> HDFS-9668 can use this to improve the locking in FsDatasetImpl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561191#comment-15561191
 ] 

Hadoop QA commented on HDFS-10895:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10895 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832402/HDFS-10895-v5.patch |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 0cf8bcbf68c7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / bea004e |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17077/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch, HDFS-10895-v5.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

2016-10-09 Thread Xiaobing Zhou (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561183#comment-15561183
 ] 

Xiaobing Zhou commented on HDFS-10972:
--

v003 addressed your most recent comments, thanks [~liuml07]

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch, 
> HDFS-10972.002.patch, HDFS-10972.003.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10972) Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'

2016-10-09 Thread Xiaobing Zhou (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HDFS-10972:
-
Attachment: HDFS-10972.003.patch

> Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'
> --
>
> Key: HDFS-10972
> URL: https://issues.apache.org/jira/browse/HDFS-10972
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs, shell, test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HDFS-10972.000.patch, HDFS-10972.001.patch, 
> HDFS-10972.002.patch, HDFS-10972.003.patch
>
>
> getDatanodeInfo should be tested in admin CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread SammiChen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-10895:
-
Attachment: HDFS-10895-v5.patch

Improve the patch after offline discussion with Kai.

> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch, HDFS-10895-v5.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

2016-10-09 Thread SammiChen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-10910:
-
Comment: was deleted

(was: Holiday from Oct. 1 to Oct. 7, please expect no response. Thanks.

)

> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10987) Make Decommission less expensive when lot of blocks present.

2016-10-09 Thread Brahma Reddy Battula (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-10987:

Priority: Critical  (was: Major)

> Make Decommission less expensive when lot of blocks present.
> 
>
> Key: HDFS-10987
> URL: https://issues.apache.org/jira/browse/HDFS-10987
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Critical
> Attachments: HDFS-10987.patch
>
>
> When user want to decommission a node which having 50M blocks +,it could hold 
> the namesystem lock for long time.We've seen it is taking 36 sec+. 
> As we knew during this time, Namenode will not available... As this 
> decommission will continuosly run till all the blocks got replicated,hence 
> Namenode will unavailable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10990) Improve the logic for TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560144#comment-15560144
 ] 

Hadoop QA commented on HDFS-10990:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 70m 
35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10990 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832347/HDFS-10990.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 796e847bd517 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / bea004e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17076/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17076/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Improve the logic for TestPendingInvalidateBlock#testPendingDeletion
> 
>
> Key: HDFS-10990
> URL: https://issues.apache.org/jira/browse/HDFS-10990
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10990.001.patch
>
>
> After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
> still fails sometimes. As [~iwasakims] commented in 
>

[jira] [Updated] (HDFS-10990) Improve the logic for TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-10990:
-
Description: After HDFS-10426, the test 
{{TestPendingInvalidateBlock#testPendingDeletion}} still fails sometimes. As 
[~iwasakims] commented in 
HDFS-10426(https://issues.apache.org/jira/browse/HDFS-10426?focusedCommentId=15539826=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15539826),
 we should add a line {{DFSTestUtil.waitForReplication}} to wait for the IBR 
and not do the assert opertion first.  (was: After HDFS-10426, the test 
{{TestPendingInvalidateBlock#testPendingDeletion}} still fails sometimes. As 
[~iwasakims] commented in HDFS-10426, we should add a line 
{{DFSTestUtil.waitForReplication}} to wait for the IBR and not do the assert 
opertion first.)

> Improve the logic for TestPendingInvalidateBlock#testPendingDeletion
> 
>
> Key: HDFS-10990
> URL: https://issues.apache.org/jira/browse/HDFS-10990
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10990.001.patch
>
>
> After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
> still fails sometimes. As [~iwasakims] commented in 
> HDFS-10426(https://issues.apache.org/jira/browse/HDFS-10426?focusedCommentId=15539826=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15539826),
>  we should add a line {{DFSTestUtil.waitForReplication}} to wait for the IBR 
> and not do the assert opertion first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10426) TestPendingInvalidateBlock failed in trunk

2016-10-09 Thread Yiqun Lin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560022#comment-15560022
 ] 

Yiqun Lin commented on HDFS-10426:
--

Good analyse! Your comment looks good to me, [~iwasakims]. I have filed a new 
JIRA HDFS-10990 to make a quick fix, I hope you will not mind that. I will very 
appreciate for that if you can make a quick review. Feel free to post your 
patch if you think my patch isn't good enough.

> TestPendingInvalidateBlock failed in trunk
> --
>
> Key: HDFS-10426
> URL: https://issues.apache.org/jira/browse/HDFS-10426
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10426.001.patch, HDFS-10426.002.patch, 
> HDFS-10426.003.patch, HDFS-10426.004.patch, HDFS-10426.005.patch, 
> HDFS-10426.006.patch
>
>
> The test {{TestPendingInvalidateBlock}} failed sometimes. The stack info:
> {code}
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
> testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)
>   Time elapsed: 7.703 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92)
> {code}
> It looks that the {{invalidateBlock}} has been removed before we do the check
> {code}
> // restart NN
> cluster.restartNameNode(true);
> dfs.delete(foo, true);
> Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal());
> Assert.assertEquals(REPLICATION, cluster.getNamesystem()
> .getPendingDeletionBlocks());
> Assert.assertEquals(REPLICATION,
> dfs.getPendingDeletionBlocksCount());
> {code}
> And I look into the related configurations. I found the property 
> {{dfs.namenode.replication.interval}} was just set as 1 second in this test. 
> And after the delay time of {{dfs.namenode.startup.delay.block.deletion.sec}} 
> and the delete operation was slowly, it will cause this case. We can see the 
> stack info before, the failed test costs 7.7s more than 5+1 second.
> One way can improve this.
> * Increase the time of {{dfs.namenode.replication.interval}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10990) Improve the logic for TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-10990:
-
Attachment: HDFS-10990.001.patch

> Improve the logic for TestPendingInvalidateBlock#testPendingDeletion
> 
>
> Key: HDFS-10990
> URL: https://issues.apache.org/jira/browse/HDFS-10990
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10990.001.patch
>
>
> After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
> still fails sometimes. As [~iwasakims] commented in HDFS-10426, we should add 
> a line {{DFSTestUtil.waitForReplication}} to wait for the IBR and not do the 
> assert opertion first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10990) Improve the logic for TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-10990:
-
Status: Patch Available  (was: Open)

Attach a simple patch for this.

> Improve the logic for TestPendingInvalidateBlock#testPendingDeletion
> 
>
> Key: HDFS-10990
> URL: https://issues.apache.org/jira/browse/HDFS-10990
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
>
> After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
> still fails sometimes. As [~iwasakims] commented in HDFS-10426, we should add 
> a line {{DFSTestUtil.waitForReplication}} to wait for the IBR and not do the 
> assert opertion first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10990) Improve the logic for TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-10990:
-
Summary: Improve the logic for 
TestPendingInvalidateBlock#testPendingDeletion  (was: Improve the logic for 
method TestPendingInvalidateBlock#testPendingDeletion)

> Improve the logic for TestPendingInvalidateBlock#testPendingDeletion
> 
>
> Key: HDFS-10990
> URL: https://issues.apache.org/jira/browse/HDFS-10990
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
>
> After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
> still fails sometimes. As [~iwasakims] commented in HDFS-10426, we should add 
> a line {{DFSTestUtil.waitForReplication}} to wait for the IBR and not do the 
> assert opertion first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10990) Improve the logic for method TestPendingInvalidateBlock#testPendingDeletion

2016-10-09 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDFS-10990:


 Summary: Improve the logic for method 
TestPendingInvalidateBlock#testPendingDeletion
 Key: HDFS-10990
 URL: https://issues.apache.org/jira/browse/HDFS-10990
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor


After HDFS-10426, the test {{TestPendingInvalidateBlock#testPendingDeletion}} 
still fails sometimes. As [~iwasakims] commented in HDFS-10426, we should add a 
line {{DFSTestUtil.waitForReplication}} to wait for the IBR and not do the 
assert opertion first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10989) Cannot get last block length after namenode failover

2016-10-09 Thread zhouyingchao (JIRA) 
zhouyingchao created HDFS-10989:
---

 Summary: Cannot get last block length after namenode failover
 Key: HDFS-10989
 URL: https://issues.apache.org/jira/browse/HDFS-10989
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: zhouyingchao


On a 2.4 cluster, access to a file failed since the last block length cannot be 
gotten.  The fsck output of the file at the moment of failure was like this:
/user/X 483600487 bytes, 2 block(s), OPENFORWRITE:  MISSING 1 blocks of 
total size 215165031 B
0. BP-219149063-10.108.84.25-1446859315800:blk_2102504098_1035525341 
len=268435456 repl=3 [10.112.17.43:11402, 10.118.22.46:11402, 
10.118.22.49:11402]
1. 
BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036219054{blockUCState=UNDER_RECOVERY,
 primaryNodeIndex=2, 
replicas=[ReplicaUnderConstruction[[DISK]DS-60be75ad-e4a7-4b1e-b3aa-327c85331d42:NORMAL|RBW],
 
ReplicaUnderConstruction[[DISK]DS-184a1ce9-655a-4e67-b0cc-29ab9984bd0a:NORMAL|RBW],
 
ReplicaUnderConstruction[[DISK]DS-6d037ac8-4bcc-4cdc-a803-55b1817e0200:NORMAL|RBW]]}
 len=215165031 MISSING!  Recorded locations [10.114.10.14:11402, 
10.118.29.3:11402, 10.118.22.42:11402]

>From those three data nodes, we found that there were IOException related to 
>the block and there were pipeline recreating events.

We figured out that there was a namenode failover event before the issue 
happened, and there were some updatePipeline calls to the earlier active 
namenode:
2016-09-27,15:04:36,437 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
updatePipeline(block=BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036137092,
 newGenerationStamp=1036170430, newLength=2624000, 
newNodes=[10.118.22.42:11402, 10.118.22.49:11402, 10.118.24.3:11402], 
clientName=DFSClient_NONMAPREDUCE_-442153643_1)
2016-09-27,15:04:36,438 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
updatePipeline(BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036137092)
 successfully to 
BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036170430
2016-09-27,15:10:10,596 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
updatePipeline(block=BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036170430,
 newGenerationStamp=1036219054, newLength=17138265, 
newNodes=[10.118.22.49:11402, 10.118.24.3:11402, 10.114.6.45:11402], 
clientName=DFSClient_NONMAPREDUCE_-442153643_1)
2016-09-27,15:10:10,601 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
updatePipeline(BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036170430)
 successfully to 
BP-219149063-10.108.84.25-1446859315800:blk_2103114087_1036219054

Whereas these new data nodes did not show up in the fsck output. It looks like 
that when data node recovers pipeline (PIPELINE_SETUP_STREAMING_RECOVERY ), the 
new data nodes would not call notifyNamingnodeReceivingBlock for the transfered 
block. 

>From code review, the issue also exists in more recent branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10988) Refactor TestBalancerBandwidth

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559712#comment-15559712
 ] 

Hadoop QA commented on HDFS-10988:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 
0 new + 9 unchanged - 5 fixed = 9 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m  6s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10988 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832332/HDFS-10988-002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cfd19e5a84e2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4d10621 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17075/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17075/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17075/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Refactor TestBalancerBandwidth
> --
>
> Key: HDFS-10988
> URL: https://issues.apache.org/jira/browse/HDFS-10988
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy

[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding doc to add how to use ISA-L based coder

2016-10-09 Thread Kai Zheng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HDFS-10895:
-
Summary: Update HDFS Erasure Coding doc to add how to use ISA-L based coder 
 (was: Update HDFS Erasure Coding document about how to enable ISA-L library 
for erasure code codec)

> Update HDFS Erasure Coding doc to add how to use ISA-L based coder
> --
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10988) Refactor TestBalancerBandwidth

2016-10-09 Thread Brahma Reddy Battula (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-10988:

Attachment: HDFS-10988-002.patch

Uploaded the patch to fix the check-style.

> Refactor TestBalancerBandwidth
> --
>
> Key: HDFS-10988
> URL: https://issues.apache.org/jira/browse/HDFS-10988
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-10988-002.patch, HDFS-10988.patch
>
>
> This jira will deal the following.
> 1) Remove Fixed sleep
> 2) Remove unused dnproxy
> 3) use try with resources



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559408#comment-15559408
 ] 

Hadoop QA commented on HDFS-10895:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  8m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10895 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832316/HDFS-10895-v4.patch |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 2294b5ecd2d5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4d10621 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17074/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Update HDFS Erasure Coding document about how to enable ISA-L library for 
> erasure code codec
> 
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec

2016-10-09 Thread SammiChen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-10895:
-
Attachment: HDFS-10895-v4.patch

remove whitespaces

> Update HDFS Erasure Coding document about how to enable ISA-L library for 
> erasure code codec
> 
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch, HDFS-10895-v4.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec

2016-10-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559377#comment-15559377
 ] 

Hadoop QA commented on HDFS-10895:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10895 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832314/HDFS-10895-v3.patch |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 66f101a55cce 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4d10621 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17073/artifact/patchprocess/whitespace-eol.txt
 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17073/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Update HDFS Erasure Coding document about how to enable ISA-L library for 
> erasure code codec
> 
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10895) Update HDFS Erasure Coding document about how to enable ISA-L library for erasure code codec

2016-10-09 Thread SammiChen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-10895:
-
Attachment: HDFS-10895-v3.patch

Improve the document by leverage "Build instructions for Hadoop"(BUILDING.txt)

> Update HDFS Erasure Coding document about how to enable ISA-L library for 
> erasure code codec
> 
>
> Key: HDFS-10895
> URL: https://issues.apache.org/jira/browse/HDFS-10895
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-10895-v1.patch, HDFS-10895-v2.patch, 
> HDFS-10895-v3.patch
>
>
> Recently, native ISA-L library support is added into HDFS erasure code RS 
> codec.  With ISA-L, the performance of erasure code RS encode/decode will be 
> significantly improved. This task is to add the deploy process into the HDFS 
> Erasure Code Document, make it easy for HDFS user to use ISA-L enabled 
> erasure code policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org