[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704573#comment-15704573
 ] 

Andrew Wang commented on HDFS-11156:


Hi folks, I'm -1 on this change. Our users rely on WebHDFS in particular for 
compatibility across major versions. In general, we should not break wire 
compatibility unless there are *very* good reasons, and we're trying to 
preserve wire compatibility even for Hadoop 3.

WebHDFS also does not expose the FileSystem interface, it exposes the HDFS 
interface. This means we sometimes return extra stuff like what we have in 
LocatedBlock vs the generic BlockLocation. This explains the mismatch with the 
FileSystem API.

I'd like to close this as WONTFIX if you agree, thanks.

> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch, HDFS-11156.04.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11181) Fuse wrapper has a typo

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704561#comment-15704561
 ] 

Hadoop QA commented on HDFS-11181:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
4s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
11s{color} | {color:green} The patch generated 0 new + 116 unchanged - 1 fixed 
= 116 total (was 117) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
14s{color} | {color:green} hadoop-hdfs-native-client in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m  9s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11181 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840660/HDFS-11181.001.patch |
| Optional Tests |  asflicense  mvnsite  unit  shellcheck  shelldocs  |
| uname | Linux 75914d84c051 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 67d9f28 |
| shellcheck | v0.4.5 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17696/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: 
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17696/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fuse wrapper has a typo
> ---
>
> Key: HDFS-11181
> URL: https://issues.apache.org/jira/browse/HDFS-11181
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Attachments: HDFS-11181.001.patch
>
>
> There's a typo in the wrapper script:
> {code}
> done < <(find "$HADOOP_HOME/hhadoop-hdfs-project" -name "*.jar" -print0)
> {code}
> It should be {{hadoop-hdfs-project}} instead of {{hhadoop-hdfs-project}}
> I regret I didn't catch the typo when I updated the wrapper last time. But I 
> don't think anyone uses the wrapper unmodified to be honest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704550#comment-15704550
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 121 unchanged - 0 fixed = 122 total (was 121) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 33s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 01cabfd8e95f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 67d9f28 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17692/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17692/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17692/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17692/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.

[jira] [Updated] (HDFS-11172) Support an erasure coding policy RS-DEFAULT-10-4-64k in HDFS

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-11172:
---
Affects Version/s: 3.0.0-alpha1
 Target Version/s: 3.0.0-alpha2
  Component/s: erasure-coding

> Support an erasure coding policy RS-DEFAULT-10-4-64k in HDFS
> 
>
> Key: HDFS-11172
> URL: https://issues.apache.org/jira/browse/HDFS-11172
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: SammiChen
>Assignee: Wei Zhou
>
> So far, "hdfs erasurecode" command supports three policies, 
> RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This task is 
> going to add RS-DEFAULT-10-4-64k policy to this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11172) Support an erasure coding policy RS-DEFAULT-10-4-64k in HDFS

[SammiChen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-11172:
-
Assignee: Wei Zhou  (was: SammiChen)

> Support an erasure coding policy RS-DEFAULT-10-4-64k in HDFS
> 
>
> Key: HDFS-11172
> URL: https://issues.apache.org/jira/browse/HDFS-11172
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: SammiChen
>Assignee: Wei Zhou
>
> So far, "hdfs erasurecode" command supports three policies, 
> RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This task is 
> going to add RS-DEFAULT-10-4-64k policy to this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11072) Add ability to unset and change directory EC policy

[SammiChen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-11072:
-
Attachment: HDFS-11072-v3.patch

1. fix style issues reported
2. add erasurecode CLI test  case
3. add more Java test case
4. double check failed test case
5. Update document HDFSErasureCoding.md about how to use new "replicate" policy

> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, 
> HDFS-11072-v2.patch, HDFS-11072-v3.patch
>
>
> Since the directory-level EC policy simply applies to files at create time, 
> it makes sense to make it more similar to storage policies and allow changing 
> and unsetting the policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11181) Fuse wrapper has a typo

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704503#comment-15704503
 ] 

Andrew Wang commented on HDFS-11181:


+1 thanks Wei-Chiu

> Fuse wrapper has a typo
> ---
>
> Key: HDFS-11181
> URL: https://issues.apache.org/jira/browse/HDFS-11181
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Attachments: HDFS-11181.001.patch
>
>
> There's a typo in the wrapper script:
> {code}
> done < <(find "$HADOOP_HOME/hhadoop-hdfs-project" -name "*.jar" -print0)
> {code}
> It should be {{hadoop-hdfs-project}} instead of {{hhadoop-hdfs-project}}
> I regret I didn't catch the typo when I updated the wrapper last time. But I 
> don't think anyone uses the wrapper unmodified to be honest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11181) Fuse wrapper has a typo

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-11181:
---
Status: Patch Available  (was: Open)

> Fuse wrapper has a typo
> ---
>
> Key: HDFS-11181
> URL: https://issues.apache.org/jira/browse/HDFS-11181
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Attachments: HDFS-11181.001.patch
>
>
> There's a typo in the wrapper script:
> {code}
> done < <(find "$HADOOP_HOME/hhadoop-hdfs-project" -name "*.jar" -print0)
> {code}
> It should be {{hadoop-hdfs-project}} instead of {{hhadoop-hdfs-project}}
> I regret I didn't catch the typo when I updated the wrapper last time. But I 
> don't think anyone uses the wrapper unmodified to be honest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11186) [SPS]: Daemon thread of SPS starts only in Active NN

[Wei Zhou (JIRA)]

Wei Zhou created HDFS-11186:
---

 Summary: [SPS]: Daemon thread of SPS starts only in Active NN
 Key: HDFS-11186
 URL: https://issues.apache.org/jira/browse/HDFS-11186
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Wei Zhou
Assignee: Wei Zhou


As discussed in [HDFS-10885 |https://issues.apache.org/jira/browse/HDFS-10885], 
we need to ensure that SPS is started only in Active NN. This JIRA is opened 
for discussion and tracking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10675) Datanode support to read from external stores.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704464#comment-15704464
 ] 

Hadoop QA commented on HDFS-10675:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
18s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
36s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
42s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
29s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
44s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
41s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
47s{color} | {color:green} HDFS-9806 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 38s{color} | {color:orange} root: The patch generated 25 new + 943 unchanged 
- 3 fixed = 968 total (was 946) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
39s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m 
39s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10675 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840804/HDFS-10675-HDFS-9806.003.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  xml  |
| uname | Linux fd30bbd94430 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-9806 / 1eae719 |
| Default Java | 1.8.0_111 |

[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704460#comment-15704460
 ] 

SammiChen commented on HDFS-11072:
--

Hi Rakesh,  sorry,  I don't know you created HDFS-11082 for "replicated" 
policy,  I originally throught you are going to fire a JIRA for 
 bq. It might also be nice to provide a CLI tool that prints the policies on 
the parent directories of a path.

After some consideration, I think it's hard to implement the "unset" function 
without the "replicated" policy. So it's better to have the "replicated" policy 
implemented in this JIRA. 
And yes, I can take HDFS-11082 if that's OK. 
 




> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, 
> HDFS-11072-v2.patch
>
>
> Since the directory-level EC policy simply applies to files at create time, 
> it makes sense to make it more similar to storage policies and allow changing 
> and unsetting the policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11184) Ozone: SCM: Make SCM use container protocol

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-11184:

Status: Patch Available  (was: Open)

> Ozone: SCM: Make SCM use container protocol
> ---
>
> Key: HDFS-11184
> URL: https://issues.apache.org/jira/browse/HDFS-11184
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-11184-HDFS-7240.001.patch
>
>
> SCM will start using container protocol to communicate with datanodes. 
> This change introduces some test failures due to some missing features which 
> will be moved to KSM. Will file separate JIRA to track disabled ozone tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11184) Ozone: SCM: Make SCM use container protocol

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-11184:

Attachment: HDFS-11184-HDFS-7240.001.patch

> Ozone: SCM: Make SCM use container protocol
> ---
>
> Key: HDFS-11184
> URL: https://issues.apache.org/jira/browse/HDFS-11184
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-11184-HDFS-7240.001.patch
>
>
> SCM will start using container protocol to communicate with datanodes. 
> This change introduces some test failures due to some missing features which 
> will be moved to KSM. Will file separate JIRA to track disabled ozone tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated HDFS-11156:
---
Attachment: HDFS-11156.04.patch

> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch, HDFS-11156.04.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11184) Ozone: SCM: Make SCM use container protocol

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704440#comment-15704440
 ] 

Anu Engineer commented on HDFS-11184:
-

Filed HDFS-11185 to track tests for ozone.

> Ozone: SCM: Make SCM use container protocol
> ---
>
> Key: HDFS-11184
> URL: https://issues.apache.org/jira/browse/HDFS-11184
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
>
> SCM will start using container protocol to communicate with datanodes. 
> This change introduces some test failures due to some missing features which 
> will be moved to KSM. Will file separate JIRA to track disabled ozone tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11185) Ozone: Fix disabled tests

[Anu Engineer (JIRA)]

Anu Engineer created HDFS-11185:
---

 Summary: Ozone: Fix disabled tests
 Key: HDFS-11185
 URL: https://issues.apache.org/jira/browse/HDFS-11185
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Anu Engineer
Assignee: Anu Engineer
 Fix For: HDFS-7240


This jira tracks disabling of some ozone tests in HDFS-11184. This is allows us 
to separate SCM and KSM into clean modules.

Disabled tests for time being are:
* testLocationsForSingleKey
* testLocationsForMultipleKeys
* testMultipleDataNodes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11184) Ozone: SCM: Make SCM use container protocol

[Anu Engineer (JIRA)]

Anu Engineer created HDFS-11184:
---

 Summary: Ozone: SCM: Make SCM use container protocol
 Key: HDFS-11184
 URL: https://issues.apache.org/jira/browse/HDFS-11184
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Anu Engineer
Assignee: Anu Engineer
 Fix For: HDFS-7240


SCM will start using container protocol to communicate with datanodes. 

This change introduces some test failures due to some missing features which 
will be moved to KSM. Will file separate JIRA to track disabled ozone tests. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11072) Add ability to unset and change directory EC policy

[Rakesh R (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704419#comment-15704419
 ] 

Rakesh R edited comment on HDFS-11072 at 11/29/16 6:49 AM:
---

Thanks [~Sammi] for the patch. Sometime back I had created HDFS-11082 to 
address the {{replicated EC policy}} case separately. Can we separate out these 
changes from this jira patch and push via HDFS-11082. I think, that would help 
to do more focused code reviews and get chance to add more unit test cases. Any 
thoughts? Also, please feel free to assign HDFS-11082 jira to you.


was (Author: rakeshr):
Thanks [~Sammi] for the patch. Sometime back I had created HDFS-11082 to 
address the {{replicated EC policy}} case separately. Can we separate out these 
changes from this jira patch and push via HDFS-11082 jira. I think that would 
help to do more focused code reviews and get chance to add more unit test 
cases. Any thoughts?

> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, 
> HDFS-11072-v2.patch
>
>
> Since the directory-level EC policy simply applies to files at create time, 
> it makes sense to make it more similar to storage policies and allow changing 
> and unsetting the policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy

[Rakesh R (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704419#comment-15704419
 ] 

Rakesh R commented on HDFS-11072:
-

Thanks [~Sammi] for the patch. Sometime back I had created HDFS-11082 to 
address the {{replicated EC policy}} case separately. Can we separate out these 
changes from this jira patch and push via HDFS-11082 jira. I think that would 
help to do more focused code reviews and get chance to add more unit test 
cases. Any thoughts?

> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, 
> HDFS-11072-v2.patch
>
>
> Since the directory-level EC policy simply applies to files at create time, 
> it makes sense to make it more similar to storage policies and allow changing 
> and unsetting the policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance

[Nandakumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704378#comment-15704378
 ] 

Nandakumar commented on HDFS-10206:
---

bq. NetworkTopology can be used by HDFS, YARN and MAPREDUCE. It is better to 
make variable names more general.
Can you point out the variables which are to be made more generic? 

bq. But the reader should pick the closest one, either "Same Node" and 
"DataNode in same rack". Perhaps you can clarify the setup.

Benchmarking was done with default replication factor - 3 (two replicas will be 
in same rack as the writer and one will be in a different rack datanode)
{{NetworkTopology.sortByDistance}} method will call 
{{NetworkTopology.getWeight}} for every replica of the block (within 
activeLen). Out of three, at least one of the replica will be in off rack 
datanode even for "Same Node" and "DataNode in same rack". With 000.patch the 
weight is calculated using network location for off rack datanodes which 
impacts the micro-benchmark results.
Sorry if I have confused you more.

{quote}
So the weight value definition has changed. It should be fine given it isn't a 
public interface. Still NetworkTopologyWithNodeGroup has its own getWeight 
definition based on the old definition. Either we update that or keep the 
weight value.
{quote}

According to {{NetworkTopologyWithNodeGroup.getWeight}}

0 for same node
1 for same group
2 for same rack
3 for off rack 

it aligns with weight definition of this patch, with an additional intermediate 
level (1 for same group)

0 for same node
2 for same rack

for off rack in {{NetworkTopologyWithNodeGroup.getWeight}} we can call 
{{super.getWeight}} which will calculate the weight using new logic rather than 
returning 3 for all the off rack nodes.



> getBlockLocations might not sort datanodes properly by distance
> ---
>
> Key: HDFS-10206
> URL: https://issues.apache.org/jira/browse/HDFS-10206
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Nandakumar
> Attachments: HDFS-10206.000.patch, HDFS-10206.001.patch, 
> HDFS-10206.002.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some 
> datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} 
> might not put the local-rack datanodes at the beginning of the sorted list. 
> That is because the function didn't call {{networktopology.add(client);}} to 
> properly set the node's parent node; something required by 
> {{networktopology.sortByDistance}} to compute distance between two nodes in 
> the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only 
> distinguishes local rack from remote rack, but it doesn't support general 
> distance calculation to tell how remote the rack is.
> {noformat}
> NetworkTopology.java
>   protected int getWeight(Node reader, Node node) {
> // 0 is local, 1 is same rack, 2 is off rack
> // Start off by initializing to off rack
> int weight = 2;
> if (reader != null) {
>   if (reader.equals(node)) {
> weight = 0;
>   } else if (isOnSameRack(reader, node)) {
> weight = 1;
>   }
> }
> return weight;
>   }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to 
> address another issue. Regardless of where we do the sorting, we still need 
> fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object 
> used by DatanodeManager and requires Nodes stored in the topology to be 
> {{DatanodeDescriptor}} for block placement. So we need to make sure we don't 
> pollute the  NetworkTopology if we plan to fix it on the server side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned

[Rakesh R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh R updated HDFS-11164:

Attachment: HDFS-11164-03.patch

> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch, HDFS-11164-01.patch, 
> HDFS-11164-02.patch, HDFS-11164-03.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will 
> internally hits the following IOException and mark the block movement as 
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it 
> will continue moving this block until it reaches {{retryMaxAttempts}}. If the 
> block movement failure(s) are only due to block pinning, then retry is 
> unnecessary. The idea of this jira is to avoid retry attempts of pinned 
> blocks as they won't be able to move to a different node. 
> {code}
> 2016-11-22 10:56:10,537 WARN 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move 
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to 
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock 
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received 
> exception java.io.IOException: Got error, status=ERROR, status message Not 
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy 
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from 
> /127.0.0.1:19501, reportedBlock move is failed
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
>   at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
>   at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
>   at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
>   at 
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.10.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch, HDFS-11149.07.patch, HDFS-11149.08.patch, 
> HDFS-11149.09.patch, HDFS-11149.10.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704298#comment-15704298
 ] 

Hadoop QA commented on HDFS-10885:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
36s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
28s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
30s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} HDFS-10285 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 
1015 unchanged - 2 fixed = 1015 total (was 1017) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
3s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 41s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.TestSafeModeWithStripedFile |
|   | hadoop.hdfs.TestMaintenanceState |
|   | hadoop.hdfs.TestFileChecksum |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10885 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840792/HDFS-10885-HDFS-10285.09.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  xml  |
| uname | Linux cf651a92c3c0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Takanobu Asanuma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704280#comment-15704280
 ] 

Takanobu Asanuma commented on HDFS-11179:
-

I'm sorry but my assumption that LightWeightHashSet has bug may not be valid. I 
will look into this more.

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Takanobu Asanuma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma updated HDFS-11179:

Priority: Major  (was: Blocker)

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Takanobu Asanuma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704276#comment-15704276
 ] 

Takanobu Asanuma commented on HDFS-11179:
-

Thanks for the comments, [~jojochuang], [~djp].

That's right, I have just observed {{postponedMisreplicatedBlocksCount}} has 
been going below zero at the moment. I'd like to set lower priority.

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704231#comment-15704231
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 121 unchanged - 0 fixed = 122 total (was 121) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
44s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 39s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 26s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux f5ae2f4cc268 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 47ca9e2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17689/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17689/artifact/patchprocess/whitespace-eol.txt
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17689/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17689/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test 

[jira] [Updated] (HDFS-10675) Datanode support to read from external stores.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-10675:
--
Status: Patch Available  (was: Open)

> Datanode support to read from external stores. 
> ---
>
> Key: HDFS-10675
> URL: https://issues.apache.org/jira/browse/HDFS-10675
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-10675-HDFS-9806.001.patch, 
> HDFS-10675-HDFS-9806.002.patch, HDFS-10675-HDFS-9806.003.patch
>
>
> This JIRA introduces a new {{PROVIDED}} {{StorageType}} to represent external 
> stores, along with enabling the Datanode to read from such stores using a 
> {{ProvidedReplica}} and a {{ProvidedVolume}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10675) Datanode support to read from external stores.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-10675:
--
Attachment: HDFS-10675-HDFS-9806.003.patch

Updated patch to fix failing test cases, and previous QA errors. 

> Datanode support to read from external stores. 
> ---
>
> Key: HDFS-10675
> URL: https://issues.apache.org/jira/browse/HDFS-10675
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-10675-HDFS-9806.001.patch, 
> HDFS-10675-HDFS-9806.002.patch, HDFS-10675-HDFS-9806.003.patch
>
>
> This JIRA introduces a new {{PROVIDED}} {{StorageType}} to represent external 
> stores, along with enabling the Datanode to read from such stores using a 
> {{ProvidedReplica}} and a {{ProvidedVolume}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704110#comment-15704110
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 27s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux d8667869e95b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 47ca9e2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17688/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17688/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>

[jira] [Updated] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on

[Wei Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zhou updated HDFS-10885:

Attachment: HDFS-10885-HDFS-10285.09.patch

Thanks [~rakeshr] for the suggestions! The failures reported are not caused by 
this patch.

> [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier 
> is on
> --
>
> Key: HDFS-10885
> URL: https://issues.apache.org/jira/browse/HDFS-10885
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: HDFS-10285
>
> Attachments: HDFS-10800-HDFS-10885-00.patch, 
> HDFS-10800-HDFS-10885-01.patch, HDFS-10800-HDFS-10885-02.patch, 
> HDFS-10885-HDFS-10285.03.patch, HDFS-10885-HDFS-10285.04.patch, 
> HDFS-10885-HDFS-10285.05.patch, HDFS-10885-HDFS-10285.06.patch, 
> HDFS-10885-HDFS-10285.07.patch, HDFS-10885-HDFS-10285.08.patch, 
> HDFS-10885-HDFS-10285.09.patch
>
>
> These two can not work at the same time to avoid conflicts and fight with 
> each other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704052#comment-15704052
 ] 

Hadoop QA commented on HDFS-11182:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 11 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 27 new + 533 unchanged - 13 fixed = 560 total (was 546) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
37s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 3 new + 7 
unchanged - 0 fixed = 10 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 13s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 13s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Private method 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.addVolumeFailureInfo(FsVolumeImpl)
 is never called  At FsVolumeList.java:called  At FsVolumeList.java:[lines 
353-357] |
| Failed junit tests | hadoop.hdfs.TestRead |
|   | hadoop.hdfs.TestPread |
|   | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
|   | hadoop.hdfs.TestDFSStripedInputStream |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
|   | hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
|   | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
|   | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
|   | hadoop.hdfs.TestSetrepIncreasing |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
|   | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
|   | hadoop.hdfs.TestInjectionForSimulatedStorage |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
|   | hadoop.hdfs.TestDecommission |
|   | 

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.09.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch, HDFS-11149.07.patch, HDFS-11149.08.patch, 
> HDFS-11149.09.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703974#comment-15703974
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m  5s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux ec13e7924035 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / d60a60b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17686/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17686/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17686/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
> 

[jira] [Commented] (HDFS-11183) Document XOR policy XOR-2-1-64k support in HDFS

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703966#comment-15703966
 ] 

SammiChen commented on HDFS-11183:
--

Thanks Andrew, I will fill the affects and targets versions later If I create 
new JIRA. Just one question. 

> Document XOR policy XOR-2-1-64k support in HDFS
> ---
>
> Key: HDFS-11183
> URL: https://issues.apache.org/jira/browse/HDFS-11183
> Project: Hadoop HDFS
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha2
>Reporter: SammiChen
>Assignee: SammiChen
>
> A new EC policy XOR-2-1-64k is added in HDFS. This task is to update HDFS 
> related documents about how to use this policy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11106) libhdfs++: Some refactoring to better organize files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703964#comment-15703964
 ] 

Hadoop QA commented on HDFS-11106:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 
57s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
31s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
14s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
0s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  8m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m  
2s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with 
JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-11106 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840753/HDFS-11106.HDFS-8707.000.patch
 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux 24b8b415a374 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-8707 / c252ac2 |
| Default Java | 1.7.0_121 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_111 
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 |
| JDK v1.7.0_121  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17685/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: 
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17685/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> libhdfs++: Some refactoring to better organize files
> 
>
> Key: HDFS-11106
> URL: https://issues.apache.org/jira/browse/HDFS-11106
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: 

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.08.patch

The v08 patch adds throttling and a few more unit tests.

This is ready for code review now.

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch, HDFS-11149.07.patch, HDFS-11149.08.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11183) Document XOR policy XOR-2-1-64k support in HDFS

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-11183:
---
Affects Version/s: 3.0.0-alpha2
 Target Version/s: 3.0.0-alpha2

Thanks for filing this Sammi, setting affects and targets versions. Please do 
this in the future, it's nice for tracking and watchers.

> Document XOR policy XOR-2-1-64k support in HDFS
> ---
>
> Key: HDFS-11183
> URL: https://issues.apache.org/jira/browse/HDFS-11183
> Project: Hadoop HDFS
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha2
>Reporter: SammiChen
>Assignee: SammiChen
>
> A new EC policy XOR-2-1-64k is added in HDFS. This task is to update HDFS 
> related documents about how to use this policy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work started] (HDFS-10860) Switch HttpFS to from Tomcat to Jetty

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-10860 started by John Zhuge.
-
> Switch HttpFS to from Tomcat to Jetty
> -
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are 
> other good options, I would propose switching to {{Jetty 9}} for the 
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet 
> Containers}}, so we don't have to change client code that much. It would 
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10860) Switch HttpFS to from Tomcat to Jetty

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10860:
--
Summary: Switch HttpFS to from Tomcat to Jetty  (was: Switch HttpFS to use 
Jetty)

> Switch HttpFS to from Tomcat to Jetty
> -
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are 
> other good options, I would propose switching to {{Jetty 9}} for the 
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet 
> Containers}}, so we don't have to change client code that much. It would 
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10860) Switch HttpFS to use Jetty

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10860:
--
Hadoop Flags: Incompatible change

> Switch HttpFS to use Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are 
> other good options, I would propose switching to {{Jetty 9}} for the 
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet 
> Containers}}, so we don't have to change client code that much. It would 
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703890#comment-15703890
 ] 

Mingliang Liu edited comment on HDFS-11094 at 11/29/16 2:18 AM:


Hi [~ebadger], thanks for updating the patch. Sorry for returning late from 
holiday. The patch looks good to me overall. I have two thoughts for your 
consideration.

# I discussed with [~arpitagarwal] offline and he suggested us use the same 
logic in {{updateActorStatesFromHeartbeat}} to update the active NN 
{{bpServiceToActive}}, which has dealt with several cases carefully. Moreover, 
if we are updating {{bpServiceToActive}} we should likely also update 
{{lastActiveClaimTxId}}. To achieve this, I think we can pass 
{{NNHAStatusHeartbeatProto}} instead of {{HAServiceStateProto}} in 
{{NamespaceInfoProto}}.
# For the unit test, can we set a very large heartbeat interval in 
configuration, and check the active NN is not null after 
{{cluster.waitForActive()}}? Mocked tests are useful as well and can be kept. 
Another idea is to drop heartbeat request against a spied HeartbeatManager.


was (Author: liuml07):
Hi [~ebadger], thanks for updating the patch. Sorry for returning late from 
holiday. The patch looks good to me overall. I have two thoughts for your 
consideration.

# I discussed with [~arpitagarwal] and he suggest us consider using the same 
logic in {{updateActorStatesFromHeartbeat}} to update the active NN 
{{bpServiceToActive}}, which has dealt with several cases carefully. If we are 
updating {{bpServiceToActive}} we should likely also update 
{{lastActiveClaimTxId}}.

> Send back HAState along with NamespaceInfo during a versionRequest as an 
> optional parameter
> ---
>
> Key: HDFS-11094
> URL: https://issues.apache.org/jira/browse/HDFS-11094
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, 
> HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, 
> HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch, 
> HDFS-11094.009.patch
>
>
> The datanode should know which NN is active when it is connecting/registering 
> to the NN. Currently, it only figures this out during its first (and 
> subsequent) heartbeat(s) and so there is a period of time where the datanode 
> is alive and registered, but can't actually do anything because it doesn't 
> know which NN is active. A byproduct of this is that the MiniDFSCluster will 
> become active before it knows what NN is active, which can lead to NPEs when 
> calling getActiveNN(). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HDFS-11072) Add ability to unset and change directory EC policy

[SammiChen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen updated HDFS-11072:
-
Comment: was deleted

(was: 1. fix style issues
2. add more test case
3. solve failed test case)

> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11072-v1.patch, HDFS-11072-v2.patch, 
> HDFS-11072-v2.patch
>
>
> Since the directory-level EC policy simply applies to files at create time, 
> it makes sense to make it more similar to storage policies and allow changing 
> and unsetting the policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10994) Support an XOR policy XOR-2-1-64k in HDFS

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703898#comment-15703898
 ] 

SammiChen commented on HDFS-10994:
--

Thanks Andrew for the remider! I created JIRA HDFS-11183 to track it. 

> Support an XOR policy XOR-2-1-64k in HDFS
> -
>
> Key: HDFS-10994
> URL: https://issues.apache.org/jira/browse/HDFS-10994
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: SammiChen
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10994-v1.patch, HDFS-10994-v2.patch, 
> HDFS-10994-v3.patch, HDFS-10994-v4.patch, HDFS-10994-v5.patch
>
>
> So far, "hdfs erasurecode" command supports three policies, 
> RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This task is 
> going to add XOR-2-1-64k policy to this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11183) Document XOR policy XOR-2-1-64k support in HDFS

[SammiChen (JIRA)]

SammiChen created HDFS-11183:


 Summary: Document XOR policy XOR-2-1-64k support in HDFS
 Key: HDFS-11183
 URL: https://issues.apache.org/jira/browse/HDFS-11183
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: SammiChen
Assignee: SammiChen


A new EC policy XOR-2-1-64k is added in HDFS. This task is to update HDFS 
related documents about how to use this policy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703890#comment-15703890
 ] 

Mingliang Liu commented on HDFS-11094:
--

Hi [~ebadger], thanks for updating the patch. Sorry for returning late from 
holiday. The patch looks good to me overall. I have two thoughts for your 
consideration.

# I discussed with [~arpitagarwal] and he suggest us consider using the same 
logic in {{updateActorStatesFromHeartbeat}} to update the active NN 
{{bpServiceToActive}}, which has dealt with several cases carefully. If we are 
updating {{bpServiceToActive}} we should likely also update 
{{lastActiveClaimTxId}}.

> Send back HAState along with NamespaceInfo during a versionRequest as an 
> optional parameter
> ---
>
> Key: HDFS-11094
> URL: https://issues.apache.org/jira/browse/HDFS-11094
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, 
> HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, 
> HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch, 
> HDFS-11094.009.patch
>
>
> The datanode should know which NN is active when it is connecting/registering 
> to the NN. Currently, it only figures this out during its first (and 
> subsequent) heartbeat(s) and so there is a period of time where the datanode 
> is alive and registered, but can't actually do anything because it doesn't 
> know which NN is active. A byproduct of this is that the MiniDFSCluster will 
> become active before it knows what NN is active, which can lead to NPEs when 
> calling getActiveNN(). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl

[Jingcheng Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703882#comment-15703882
 ] 

Jingcheng Du commented on HDFS-9668:


Hi [~arpitagarwal], would you mind reviewing the patch V25? thanks a lot!

> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch, 
> HDFS-9668-11.patch, HDFS-9668-12.patch, HDFS-9668-13.patch, 
> HDFS-9668-14.patch, HDFS-9668-14.patch, HDFS-9668-15.patch, 
> HDFS-9668-16.patch, HDFS-9668-17.patch, HDFS-9668-18.patch, 
> HDFS-9668-19.patch, HDFS-9668-19.patch, HDFS-9668-2.patch, 
> HDFS-9668-20.patch, HDFS-9668-21.patch, HDFS-9668-22.patch, 
> HDFS-9668-23.patch, HDFS-9668-23.patch, HDFS-9668-24.patch, 
> HDFS-9668-25.patch, HDFS-9668-3.patch, HDFS-9668-4.patch, HDFS-9668-5.patch, 
> HDFS-9668-6.patch, HDFS-9668-7.patch, HDFS-9668-8.patch, HDFS-9668-9.patch, 
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in 
> SSD/RAMDISK, and all other files are stored in HDD), we observe many 
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part 
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at 
> /192.168.50.16:48521 [Receiving block 
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread 
> t@93336
>java.lang.Thread.State: BLOCKED
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
>   - waiting to lock <18324c9> (a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by 
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at 
> /192.168.50.16:48520 [Receiving block 
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
>   at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - None
>   
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at 
> /192.168.50.16:48520 [Receiving block 
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread 
> t@93335
>java.lang.Thread.State: RUNNABLE
>   at java.io.UnixFileSystem.createFileExclusively(Native Method)
>   at java.io.File.createNewFile(File.java:1012)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
>   - locked <18324c9> (a 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
>   at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
>   at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
>   - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the 
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy 
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a 
> slow storage can block all the other same operations in the same DataNode, 
> especially in HBase when many wal/flusher/compactor are 

[jira] [Updated] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11182:
-
Status: Open  (was: Patch Available)

> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in 
> HDFS-11149 to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11182:
-
Attachment: (was: HDFS-11182.01.patch)

> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in 
> HDFS-11149 to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10930) Refactor: Wrap Datanode IO related operations

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703819#comment-15703819
 ] 

Arpit Agarwal commented on HDFS-10930:
--

+1 for the v07 patch pending Jenkins. Thanks [~xyao]!

Looks like this Jenkins run picked up the pull request. We'll need to either 
update the PR or just unlink it so Jenkins uses the attached patch file.

> Refactor: Wrap Datanode IO related operations
> -
>
> Key: HDFS-10930
> URL: https://issues.apache.org/jira/browse/HDFS-10930
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-10930.01.patch, HDFS-10930.02.patch, 
> HDFS-10930.03.patch, HDFS-10930.04.patch, HDFS-10930.05.patch, 
> HDFS-10930.06.patch, HDFS-10930.07.patch
>
>
> Datanode IO (Disk/Network) related operations and instrumentations are 
> currently spilled in many classes such as DataNode.java, BlockReceiver.java, 
> BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, 
> DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, 
> LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc. 
> This ticket is opened to consolidate IO related operations for easy 
> instrumentation, metrics collection, logging and trouble shooting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703810#comment-15703810
 ] 

Hadoop QA commented on HDFS-11180:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
59s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 51s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Inconsistent synchronization of 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.editLogStream; locked 78% of 
time  Unsynchronized access at FSEditLog.java:78% of time  Unsynchronized 
access at FSEditLog.java:[line 1779] |
|  |  Inconsistent synchronization of 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.state; locked 88% of time  
Unsynchronized access at FSEditLog.java:88% of time  Unsynchronized access at 
FSEditLog.java:[line 571] |
|  |  Increment of volatile field 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.txid in 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.beginTransaction()  At 
FSEditLog.java:in 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.beginTransaction()  At 
FSEditLog.java:[line 519] |
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11180 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840741/HDFS-11180.03.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux b4d9efddde63 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 62b42ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 

[jira] [Updated] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11182:
-
Attachment: HDFS-11182.01.patch

v01 patch for Jenkins run.

> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11182.01.patch
>
>
> Update DataNode to use the DatasetVolumeChecker class introduced in 
> HDFS-11149 to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11182:
-
Status: Patch Available  (was: Open)

> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in 
> HDFS-11149 to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703804#comment-15703804
 ] 

ASF GitHub Bot commented on HDFS-11182:
---

GitHub user arp7 opened a pull request:

https://github.com/apache/hadoop/pull/168

HDFS-11182. Update DataNode to use DatasetVolumeChecker.

Preliminary patch for Jenkins runs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arp7/hadoop HDFS-11182

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #168


commit 621fdc6118c8fe3f738ca8a86ebc9dac71804298
Author: Arpit Agarwal 
Date:   2016-11-23T19:38:14Z

HDFS-11149. Support for parallel checking of FsVolumes.

commit b9ede5b8b990821a681bc8fbc9975ee93fb158d6
Author: Arpit Agarwal 
Date:   2016-11-29T01:25:51Z

HDFS-11182. Update DataNode to use DatasetVolumeChecker.




> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in 
> HDFS-11149 to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11106) libhdfs++: Some refactoring to better organize files

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-11106:
---
Status: Patch Available  (was: Open)

> libhdfs++: Some refactoring to better organize files
> 
>
> Key: HDFS-11106
> URL: https://issues.apache.org/jira/browse/HDFS-11106
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-11106.HDFS-8707.000.patch
>
>
> I propose splitting some of the files that have grown wild over time into 
> files that align with more specific functionality.  It's probably best to do 
> this in a few pieces so it doesn't invalidate anyone's patches in progress.  
> Here's what I have in mind, looking for feedback if 1) it's not worth doing 
> for some reason 2) it will break your patch and you'd like this to wait.  I'd 
> also like to consolidate related functions, mostly protobuf helpers, that are 
> spread around the library into dedicated files. 
> Targets (can split each into a separate patch):
> * split hdfs.cc into hdfs.cc and hdfs_ext.cc.  Already have a separate 
> hdfs_ext.h for C bindings for libhdfs++ specific extensions so 
> implementations of those that live in hdfs.cc would be moved out.  Just makes 
> things a little cleaner.
> * separate the implementation of operations from async shim code in files 
> like filesystem.cc (make a filesystem_shims.cc).  The shims are just 
> boilerplate code that only need to change if the signature of their async 
> counterparts change.
> * split apart various RPC code based on classes.  Things like Request and 
> RpcConnection get defined in rpc_engine.h and then implemented in a handful 
> of files which get confusing to navigate e.g. why would one expect Request's 
> implementation to be in rpc_connection.cc.
> * Move all of the protobuf<->C++ struct conversion helpers and protobuf wire 
> serialization/deserialization functions into a single file.  Gives us less 
> protobuf header includes and less accidental duplication of these sorts of 
> functions.
> * merge base64.cc into util.cc; base64.cc only contains a single utility 
> function.
> * rename hdfs_public_api.h/cc to hdfs_ioservice.h/cc.  Originally all of the 
> implementation declarations of the public API classes like FileSystemImpl 
> were going to live in here.  Currently only the hdfs::IoServiceImpl lives in 
> there and the other Impl classes have their own dedicated files.
> Like any refactoring some of it comes down to personal preferences.  My hope 
> is that by breaking these into smaller patches/commits relatively fast 
> forward progress can be made on stuff everyone agrees while things that 
> people are concerned about can be worked out in a way that satisfies everyone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11181) Fuse wrapper has a typo

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703778#comment-15703778
 ] 

Yiqun Lin commented on HDFS-11181:
--

+1 for the fix, you forget to click the "submit patch", [~jojochuang].

> Fuse wrapper has a typo
> ---
>
> Key: HDFS-11181
> URL: https://issues.apache.org/jira/browse/HDFS-11181
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fuse-dfs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Attachments: HDFS-11181.001.patch
>
>
> There's a typo in the wrapper script:
> {code}
> done < <(find "$HADOOP_HOME/hhadoop-hdfs-project" -name "*.jar" -print0)
> {code}
> It should be {{hadoop-hdfs-project}} instead of {{hhadoop-hdfs-project}}
> I regret I didn't catch the typo when I updated the wrapper last time. But I 
> don't think anyone uses the wrapper unmodified to be honest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.07.patch

v07 patch adds a new test case (testCheckingClosedVolume).

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch, HDFS-11149.07.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11178) TestAddStripedBlockInFBR#testAddBlockInFullBlockReport fails frequently in trunk

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703765#comment-15703765
 ] 

Yiqun Lin commented on HDFS-11178:
--

Thanks [~liuml07] for taking a look.
{quote}
Is MiniDFSCluster#triggerBlockReports() blocking call?
{quote}
No, I think it's not a blocking call. {{MiniDFSCluster#triggerBlockReports()}} 
just traverse the DNs in cluster, and do the 
{{DataNodeTestUtils.triggerBlockReport(dn);}} for each DN, while 
{{DataNodeTestUtils.triggerBlockReport(dn)}} is a non-blocking call.

> TestAddStripedBlockInFBR#testAddBlockInFullBlockReport fails frequently in 
> trunk
> 
>
> Key: HDFS-11178
> URL: https://issues.apache.org/jira/browse/HDFS-11178
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11178.001.patch, HDFS-11178.002.patch
>
>
> The test {{TestAddStripedBlockInFBR#testAddBlockInFullBlockReport}} fails 
> easily in trunk. It's easy to reproduce the failure, it fails 2~3 times when 
> I run the test 4~-5 times in my local. Also it failed in the recent 
> Jenkins(https://builds.apache.org/job/PreCommit-HDFS-Build/17667/testReport/),
>  The stack infos:
> {code}
> java.lang.AssertionError: expected:<9> but was:<7>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR.testAddBlockInFullBlockReport(TestAddStripedBlockInFBR.java:108)
> {code}
> It's easy to have a fix: Use {{GenericTestUtils.waitFor}} to wait the full 
> blocks being reported. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10930) Refactor: Wrap Datanode IO related operations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703763#comment-15703763
 ] 

Hadoop QA commented on HDFS-10930:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 1s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 467 unchanged - 12 fixed = 470 total (was 479) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
37s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 7 
unchanged - 0 fixed = 8 total (was 7) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 46s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 48s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.validateIntegrityAndSetLength(File,
 long) may fail to clean up java.io.InputStream on checked exception  
Obligation to clean up resource created at BlockPoolSlice.java:clean up 
java.io.InputStream on checked exception  Obligation to clean up resource 
created at BlockPoolSlice.java:[line 718] is not discharged |
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10930 |
| GITHUB PR | https://github.com/apache/hadoop/pull/160 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7f9620b26d5a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17681/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
 |
| 

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703701#comment-15703701
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 120 unchanged - 0 fixed = 121 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 23s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
|   | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 10dcfd29ea54 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17684/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17684/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17684/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17684/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> 

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703661#comment-15703661
 ] 

Hadoop QA commented on HDFS-11149:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m 
28s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux c507890a5617 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17682/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17682/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> 

[jira] [Updated] (HDFS-11106) libhdfs++: Some refactoring to better organize files

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-11106:
---
Attachment: HDFS-11106.HDFS-8707.000.patch

First pass, added a few comments but otherwise didn't change code beyond moving 
it around.

-Separated synchronous shims from filesystem.cc into filesystem_sync.cc.  Added 
some comments to explain what the shims are really doing.

-Moved contents of base64.cc into util.cc.  Got rid of unnecessary includes.

-Renamed hdfs_public_api to hdfs_ioservice and added comments to explain what 
the exception handling was there for and why it's useful to wrap the 
asio::ioservice.

[~xiaowei.zhu] or [~swalk...@yahoo.com] could you take a look at this?  If 
these look like they'll improve maintainability to you I'll commit them and 
then make another patch that consolidates protobuf utilities and separates 
libhdfs++ specific extensions out of hdfs.cc.

> libhdfs++: Some refactoring to better organize files
> 
>
> Key: HDFS-11106
> URL: https://issues.apache.org/jira/browse/HDFS-11106
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-11106.HDFS-8707.000.patch
>
>
> I propose splitting some of the files that have grown wild over time into 
> files that align with more specific functionality.  It's probably best to do 
> this in a few pieces so it doesn't invalidate anyone's patches in progress.  
> Here's what I have in mind, looking for feedback if 1) it's not worth doing 
> for some reason 2) it will break your patch and you'd like this to wait.  I'd 
> also like to consolidate related functions, mostly protobuf helpers, that are 
> spread around the library into dedicated files. 
> Targets (can split each into a separate patch):
> * split hdfs.cc into hdfs.cc and hdfs_ext.cc.  Already have a separate 
> hdfs_ext.h for C bindings for libhdfs++ specific extensions so 
> implementations of those that live in hdfs.cc would be moved out.  Just makes 
> things a little cleaner.
> * separate the implementation of operations from async shim code in files 
> like filesystem.cc (make a filesystem_shims.cc).  The shims are just 
> boilerplate code that only need to change if the signature of their async 
> counterparts change.
> * split apart various RPC code based on classes.  Things like Request and 
> RpcConnection get defined in rpc_engine.h and then implemented in a handful 
> of files which get confusing to navigate e.g. why would one expect Request's 
> implementation to be in rpc_connection.cc.
> * Move all of the protobuf<->C++ struct conversion helpers and protobuf wire 
> serialization/deserialization functions into a single file.  Gives us less 
> protobuf header includes and less accidental duplication of these sorts of 
> functions.
> * merge base64.cc into util.cc; base64.cc only contains a single utility 
> function.
> * rename hdfs_public_api.h/cc to hdfs_ioservice.h/cc.  Originally all of the 
> implementation declarations of the public API classes like FileSystemImpl 
> were going to live in here.  Currently only the hdfs::IoServiceImpl lives in 
> there and the other Impl classes have their own dedicated files.
> Like any refactoring some of it comes down to personal preferences.  My hope 
> is that by breaking these into smaller patches/commits relatively fast 
> forward progress can be made on stuff everyone agrees while things that 
> people are concerned about can be worked out in a way that satisfies everyone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11178) TestAddStripedBlockInFBR#testAddBlockInFullBlockReport fails frequently in trunk

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703492#comment-15703492
 ] 

Mingliang Liu commented on HDFS-11178:
--

Is {{MiniDFSCluster#triggerBlockReports()}} blocking call? If so can we simply 
trigger full block reports on all DNs instead of DN0?

> TestAddStripedBlockInFBR#testAddBlockInFullBlockReport fails frequently in 
> trunk
> 
>
> Key: HDFS-11178
> URL: https://issues.apache.org/jira/browse/HDFS-11178
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11178.001.patch, HDFS-11178.002.patch
>
>
> The test {{TestAddStripedBlockInFBR#testAddBlockInFullBlockReport}} fails 
> easily in trunk. It's easy to reproduce the failure, it fails 2~3 times when 
> I run the test 4~-5 times in my local. Also it failed in the recent 
> Jenkins(https://builds.apache.org/job/PreCommit-HDFS-Build/17667/testReport/),
>  The stack infos:
> {code}
> java.lang.AssertionError: expected:<9> but was:<7>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR.testAddBlockInFullBlockReport(TestAddStripedBlockInFBR.java:108)
> {code}
> It's easy to have a fix: Use {{GenericTestUtils.waitFor}} to wait the full 
> blocks being reported. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703429#comment-15703429
 ] 

Junping Du commented on HDFS-11179:
---

Agree with [~jojochuang]. Especially, this EC staff doesn't show up in 2.8 
release, remove 2.8 in target version.

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated HDFS-11179:
--
Target Version/s: 3.0.0-alpha2  (was: 2.8.0, 3.0.0-alpha2)

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: (was: HDFS-11149.06.patch)

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.06.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8678) Bring back the feature to view chunks of files in the HDFS file browser

[Allen Wittenauer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703425#comment-15703425
 ] 

Allen Wittenauer commented on HDFS-8678:


We should probably move the block information to the bottom since it's not 
particularly useful to 99.% of the users.  But we can do that as a 
follow-up if you want.

+1

> Bring back the feature to view chunks of files in the HDFS file browser
> ---
>
> Key: HDFS-8678
> URL: https://issues.apache.org/jira/browse/HDFS-8678
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ui
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-8678.01.patch, HDFS-8678.02.patch, 
> HDFS-8678.03.patch
>
>
> The legacy file browser displayed small chunks of a file in the browser 
> itself. This was useful to users because they can quickly verify that their 
> input or output is in the format they expect. We should bring back this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.06.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: (was: HDFS-11149.06.patch)

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.06.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: (was: HDFS-11149.06.patch)

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HDFS-11180:
-
Attachment: HDFS-11180.03.patch

03 patch:
* Added a regression test
* Updated FSEditLog and some metric related code to pass the regression test
* TODO: Ignore findbugs warnings

> Intermittent deadlock in NameNode when failover happens.
> 
>
> Key: HDFS-11180
> URL: https://issues.apache.org/jira/browse/HDFS-11180
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Abhishek Modi
>Assignee: Akira Ajisaka
>Priority: Blocker
>  Labels: high-availability
> Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, 
> HDFS-11180.02.patch, HDFS-11180.03.patch, jstack.log
>
>
> It is happening due to metrics getting updated at the same time when failover 
> is happening. Please find attached jstack at that point of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.06.patch

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch, 
> HDFS-11149.06.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-5517) Lower the default maximum number of blocks per file

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703397#comment-15703397
 ] 

Hadoop QA commented on HDFS-5517:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
1s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 398 unchanged - 1 fixed = 399 total (was 399) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m  3s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 28s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-5517 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840722/HDFS-5517.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 8dcf2ac8bc4a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17680/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 

[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations

[Xiaoyu Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoyu Yao updated HDFS-10930:
--
Attachment: HDFS-10930.07.patch

Thanks [~arpitagarwal] for the review. I've updated the patch based on the 
feedback for the findbugs issue. 

> Refactor: Wrap Datanode IO related operations
> -
>
> Key: HDFS-10930
> URL: https://issues.apache.org/jira/browse/HDFS-10930
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-10930.01.patch, HDFS-10930.02.patch, 
> HDFS-10930.03.patch, HDFS-10930.04.patch, HDFS-10930.05.patch, 
> HDFS-10930.06.patch, HDFS-10930.07.patch
>
>
> Datanode IO (Disk/Network) related operations and instrumentations are 
> currently spilled in many classes such as DataNode.java, BlockReceiver.java, 
> BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java, 
> DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java, 
> LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc. 
> This ticket is opened to consolidate IO related operations for easy 
> instrumentation, metrics collection, logging and trouble shooting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703372#comment-15703372
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 8 new + 120 unchanged - 0 fixed = 128 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 29s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestFsDatasetCache |
|   | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 9008b01e708c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17679/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17679/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17679/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17679/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> 

[jira] [Commented] (HDFS-11152) Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703291#comment-15703291
 ] 

Andrew Wang commented on HDFS-11152:


Hi Sammi, thanks for the patch. Looks good overall, but are there places we 
should add Precondition checks (e.g. fsimage/editlog loading, file read and 
write) for correctness? Unit tests for these situations would also be good.

> Start erasure coding policy ID number from 1 instead of 0 to void potential 
> unexpected errors
> -
>
> Key: HDFS-11152
> URL: https://issues.apache.org/jira/browse/HDFS-11152
> Project: Hadoop HDFS
>  Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: SammiChen
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11152-v1.patch
>
>
> This task will change erasure coding policy ID number starting from 1 instead 
> of current 0, to avoid some potential unexpected errors in codes since 0 is 
> default value for integer variables. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703280#comment-15703280
 ] 

Hadoop QA commented on HDFS-11180:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
56s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 42s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Inconsistent synchronization of 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.state; locked 96% of time  
Unsynchronized access at FSEditLog.java:96% of time  Unsynchronized access at 
FSEditLog.java:[line 550] |
|  |  Increment of volatile field 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.txid in 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.beginTransaction()  At 
FSEditLog.java:in 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.beginTransaction()  At 
FSEditLog.java:[line 498] |
| Failed junit tests | 
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11180 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840665/HDFS-11180.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 05e9d2dfbf85 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a2b1ff0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17678/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
 |
| unit | 

[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703262#comment-15703262
 ] 

Akira Ajisaka commented on HDFS-11180:
--

Cancelling patch because we need to create unsynchronized 
{{getTotalSyncCount()}} as well.

> Intermittent deadlock in NameNode when failover happens.
> 
>
> Key: HDFS-11180
> URL: https://issues.apache.org/jira/browse/HDFS-11180
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Abhishek Modi
>Assignee: Akira Ajisaka
>Priority: Blocker
>  Labels: high-availability
> Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, 
> HDFS-11180.02.patch, jstack.log
>
>
> It is happening due to metrics getting updated at the same time when failover 
> is happening. Please find attached jstack at that point of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11182) Update DataNode to use DatasetVolumeChecker

[Arpit Agarwal (JIRA)]

Arpit Agarwal created HDFS-11182:


 Summary: Update DataNode to use DatasetVolumeChecker
 Key: HDFS-11182
 URL: https://issues.apache.org/jira/browse/HDFS-11182
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal


Update DataNode to use the DatasetVolumeChecker class introduced in HDFS-11149 
to parallelize disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703169#comment-15703169
 ] 

Wei-Chiu Chuang commented on HDFS-11179:


[~tasanuma0829] thanks for filing the bug report.
While I think this fix is important, I don't see why it's a blocker.

For reference, we've seen block id overflow in the past in logs, but we've not 
noticed it caused real issues.

> LightWeightHashSet can't remove blocks correctly which have a large number 
> blockId
> --
>
> Key: HDFS-11179
> URL: https://issues.apache.org/jira/browse/HDFS-11179
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Blocker
>
> Our test cluster has faced a problem that 
> {{postponedMisreplicatedBlocksCount}} has been going below zero. The version 
> of the cluster is a recent 3.0. We haven't created any EC files yet. This is 
> the NN's log:
> {noformat}
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 448 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 13 msecs. 272 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. 96 blocks are 
> left. 176 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 327 msecs. -77 blocks are 
> left. 177 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 15 msecs. -253 blocks are 
> left. 179 blocks are removed.
> Rescan of postponedMisreplicatedBlocks completed in 14 msecs. -432 blocks are 
> left. 179 blocks are removed.
> {noformat}
> I looked into this issue and found that it is caused by 
> {{LightWeightHashSet}} which is used for {{postponedMisreplicatedBlocks}} 
> recently. When {{LightWeightHashSet}} remove blocks which have a large number 
> blockId, overflows happen and the blocks can't be removed correctly(, let 
> alone ec blocks whose blockId starts with the minimum of long).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-5517) Lower the default maximum number of blocks per file

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-5517:
--
Attachment: HDFS-5517.002.patch

I did the trival rebase for this change, patch attached.

> Lower the default maximum number of blocks per file
> ---
>
> Key: HDFS-5517
> URL: https://issues.apache.org/jira/browse/HDFS-5517
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.2.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
>  Labels: BB2015-05-TBR
> Attachments: HDFS-5517.002.patch, HDFS-5517.patch
>
>
> We introduced the maximum number of blocks per file in HDFS-4305, but we set 
> the default to 1MM. In practice this limit is so high as to never be hit, 
> whereas we know that an individual file with 10s of thousands of blocks can 
> cause problems. We should lower the default value, in my opinion to 10k.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703132#comment-15703132
 ] 

Mingliang Liu commented on HDFS-11156:
--

The patch looks good to me overall. +1

I have several minor comments:

# Do we need to specially handle the case of {{list.isEmpty()}}? Looks pretty 
similar to the {{else}} clause.
{code}
593 } else if (list.isEmpty()) {
594   return new String[0];
595 } else {
596   final String[] array = new String[list.size()];
597   int i = 0;
598   for (Object object : list) {
599 array[i++] = object.toString();
600   }
601   return array;
602 }
{code}
# Are you considering using {{assertArrayEquals()}}?
{code:title=testWebHdfsGetBlockLocationsWithStorageType()}
899 for(int j=0; j Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.05.patch

v05 patch fixes checkstyle issues.

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch, HDFS-11149.05.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-6984:
--
Target Version/s: 3.0.0-alpha2  (was: )

Setting target version for tracking.

> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>  Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch, 
> HDFS-6984.003.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier.  Originally, we used this 
> to serialize it and send it over the wire.  But in Hadoop 2 and later, we 
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this 
> information.  The protobuf form is preferable, since it allows us to add new 
> fields in a backwards-compatible way.  Another issue is that already a lot of 
> subclasses of FileStatus don't override the Writable methods of the 
> superclass, breaking the interface contract that read(status.write) should be 
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so 
> that we don't have to deal with these issues.  It's probably too late to do 
> this in Hadoop 2, since user code may be relying on the existing FileStatus 
> serialization there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703055#comment-15703055
 ] 

Hadoop QA commented on HDFS-11149:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 26 new + 120 unchanged - 0 fixed = 146 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m 
17s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux dc16844ae959 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5d5614f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17677/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17677/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17677/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: 

[jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance

[Ming Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702984#comment-15702984
 ] 

Ming Ma commented on HDFS-10206:


* NetworkTopology can be used by HDFS, YARN and MAPREDUCE. It is better to make 
variable names more general.

bq. Out of three replica, one will be in off rack datanode which is causing the 
difference
But the reader should pick the closest one, either "Same Node" and "DataNode in 
same rack". Perhaps you can clarify the setup.

bq. Weight calculation after this patch
So the weight value definition has changed. It should be fine given it isn't a 
public interface. Still NetworkTopologyWithNodeGroup has its own getWeight 
definition based on the old definition. Either we update that or keep the 
weight value.



> getBlockLocations might not sort datanodes properly by distance
> ---
>
> Key: HDFS-10206
> URL: https://issues.apache.org/jira/browse/HDFS-10206
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Nandakumar
> Attachments: HDFS-10206.000.patch, HDFS-10206.001.patch, 
> HDFS-10206.002.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some 
> datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}} 
> might not put the local-rack datanodes at the beginning of the sorted list. 
> That is because the function didn't call {{networktopology.add(client);}} to 
> properly set the node's parent node; something required by 
> {{networktopology.sortByDistance}} to compute distance between two nodes in 
> the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only 
> distinguishes local rack from remote rack, but it doesn't support general 
> distance calculation to tell how remote the rack is.
> {noformat}
> NetworkTopology.java
>   protected int getWeight(Node reader, Node node) {
> // 0 is local, 1 is same rack, 2 is off rack
> // Start off by initializing to off rack
> int weight = 2;
> if (reader != null) {
>   if (reader.equals(node)) {
> weight = 0;
>   } else if (isOnSameRack(reader, node)) {
> weight = 1;
>   }
> }
> return weight;
>   }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to 
> address another issue. Regardless of where we do the sorting, we still need 
> fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object 
> used by DatanodeManager and requires Nodes stored in the topology to be 
> {{DatanodeDescriptor}} for block placement. So we need to make sure we don't 
> pollute the  NetworkTopology if we plan to fix it on the server side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HDFS-11180:
-
Priority: Blocker  (was: Major)

> Intermittent deadlock in NameNode when failover happens.
> 
>
> Key: HDFS-11180
> URL: https://issues.apache.org/jira/browse/HDFS-11180
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Abhishek Modi
>Assignee: Akira Ajisaka
>Priority: Blocker
>  Labels: high-availability
> Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, 
> HDFS-11180.02.patch, jstack.log
>
>
> It is happening due to metrics getting updated at the same time when failover 
> is happening. Please find attached jstack at that point of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HDFS-11180:
-
Assignee: Akira Ajisaka
Target Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6
  Status: Patch Available  (was: Open)

> Intermittent deadlock in NameNode when failover happens.
> 
>
> Key: HDFS-11180
> URL: https://issues.apache.org/jira/browse/HDFS-11180
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Abhishek Modi
>Assignee: Akira Ajisaka
>  Labels: high-availability
> Attachments: HDFS-11180.00.patch, HDFS-11180.01.patch, 
> HDFS-11180.02.patch, jstack.log
>
>
> It is happening due to metrics getting updated at the same time when failover 
> is happening. Please find attached jstack at that point of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11149) Support for parallel checking of FsVolumes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702928#comment-15702928
 ] 

Hadoop QA commented on HDFS-11149:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 6 new + 120 unchanged - 0 fixed = 126 total (was 120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 25s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
|   | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11149 |
| GITHUB PR | https://github.com/apache/hadoop/pull/165 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux f9b7c8a612fb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5d5614f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17676/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17676/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17676/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17676/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically 

[jira] [Commented] (HDFS-10994) Support an XOR policy XOR-2-1-64k in HDFS

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702884#comment-15702884
 ] 

Andrew Wang commented on HDFS-10994:


Nice work here Sammi! Do we also need a follow on for a documentation update?

> Support an XOR policy XOR-2-1-64k in HDFS
> -
>
> Key: HDFS-10994
> URL: https://issues.apache.org/jira/browse/HDFS-10994
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: SammiChen
>Assignee: SammiChen
>  Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10994-v1.patch, HDFS-10994-v2.patch, 
> HDFS-10994-v3.patch, HDFS-10994-v4.patch, HDFS-10994-v5.patch
>
>
> So far, "hdfs erasurecode" command supports three policies, 
> RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k and RS-LEGACY-6-3-64k. This task is 
> going to add XOR-2-1-64k policy to this command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.04.patch

v04 patch adds a test case for disk check hang.

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch, HDFS-11149.04.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on

[Rakesh R (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702726#comment-15702726
 ] 

Rakesh R commented on HDFS-10885:
-

Thanks [~zhouwei] for the good work. I'm +1 for the latest patch after 
addressing the below comments.

# Could you mention the default behavior in hdfs-default.xml something like,
{code}
If true, enables StoragePolicySatisfier. By default, StoragePolicySatisfier is 
enabled.
{code}
# Few minor [checkstyle 
warnings|https://builds.apache.org/job/PreCommit-HDFS-Build/17673/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt]
 reported, please take care.
# Just a suggstion. Its good practice to give message. Probably can use like 
below,
{code}
Assert.assertFalse("SPS should not start when a Mover instance is running", 
active);
{code}

[~umamaheswararao], could you please let me know your views about the necessity 
of [double checks ensuring that SPS/Mover is not running 
together|https://issues.apache.org/jira/browse/HDFS-10885?focusedCommentId=15648191=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15648191],
 thanks!

> [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier 
> is on
> --
>
> Key: HDFS-10885
> URL: https://issues.apache.org/jira/browse/HDFS-10885
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: HDFS-10285
>
> Attachments: HDFS-10800-HDFS-10885-00.patch, 
> HDFS-10800-HDFS-10885-01.patch, HDFS-10800-HDFS-10885-02.patch, 
> HDFS-10885-HDFS-10285.03.patch, HDFS-10885-HDFS-10285.04.patch, 
> HDFS-10885-HDFS-10285.05.patch, HDFS-10885-HDFS-10285.06.patch, 
> HDFS-10885-HDFS-10285.07.patch, HDFS-10885-HDFS-10285.08.patch
>
>
> These two can not work at the same time to avoid conflicts and fight with 
> each other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11149) Support for parallel checking of FsVolumes

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11149:
-
Attachment: HDFS-11149.03.patch

Fix issues flagged by Jenkins.

> Support for parallel checking of FsVolumes
> --
>
> Key: HDFS-11149
> URL: https://issues.apache.org/jira/browse/HDFS-11149
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-11149.01.patch, HDFS-11149.02.patch, 
> HDFS-11149.03.patch
>
>
> Similar to HDFS-9, the DataNode can parallelize checking FsVolumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11171) add metric to report client shortcircuit reads

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-11171:
-
Status: Open  (was: Patch Available)

> add metric to report client shortcircuit reads
> --
>
> Key: HDFS-11171
> URL: https://issues.apache.org/jira/browse/HDFS-11171
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.7.1
>Reporter: 5feixiang
>  Labels: metrics
> Fix For: 2.7.1
>
>
> Current dfs context metrics only contains bytesRead ,remoteBytesRead and 
> totalReadTime.We can find  that bytesRead= remoteBytesRead +balance remote 
> copyBytesRead+localBytesRead, so we can add  localBytesRead and localReadTime 
> to calcute local read rate which is usefu to compate local read rate between 
> short-circuit read and tcp-socket read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702598#comment-15702598
 ] 

Hadoop QA commented on HDFS-11072:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
11s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
26s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
1s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 56s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
|   | hadoop.cli.TestErasureCodingCLI |
|   | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11072 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840651/HDFS-11072-v2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 47f501302c0d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5d5614f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17674/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17674/testReport/ |
| modules | C: 

[jira] [Commented] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702577#comment-15702577
 ] 

Hadoop QA commented on HDFS-9483:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9483 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12840560/HDFS-9483.002.patch |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux fe53cdd5a1f0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5d5614f |
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17675/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured 
> WebHDFS.
> -
>
> Key: HDFS-9483
> URL: https://issues.apache.org/jira/browse/HDFS-9483
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9483.001.patch, HDFS-9483.002.patch, HDFS-9483.patch
>
>
> If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in 
> a URL to access it.  The current documentation does not state this anywhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702541#comment-15702541
 ] 

Brahma Reddy Battula edited comment on HDFS-9483 at 11/28/16 5:19 PM:
--

[~surendrasingh] thanks for updating the patch. changes looks fine to 
me..Triggered the jenkins..

[~cnauroth] does it needs some more update.like SSL config link.?


was (Author: brahmareddy):
[~surendrasingh] thanks for updating the patch. changes looks fine to 
me..Triggered the jenkins..

[~cnauroth] does it needs some more update..?

> Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured 
> WebHDFS.
> -
>
> Key: HDFS-9483
> URL: https://issues.apache.org/jira/browse/HDFS-9483
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9483.001.patch, HDFS-9483.002.patch, HDFS-9483.patch
>
>
> If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in 
> a URL to access it.  The current documentation does not state this anywhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702541#comment-15702541
 ] 

Brahma Reddy Battula commented on HDFS-9483:


[~surendrasingh] thanks for updating the patch. changes looks fine to 
me..Triggered the jenkins..

[~cnauroth] does it needs some more update..?

> Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured 
> WebHDFS.
> -
>
> Key: HDFS-9483
> URL: https://issues.apache.org/jira/browse/HDFS-9483
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Chris Nauroth
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9483.001.patch, HDFS-9483.002.patch, HDFS-9483.patch
>
>
> If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in 
> a URL to access it.  The current documentation does not state this anywhere.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org