date:20161013

[jira] [Updated] (HADOOP-13417) Fix javac and checkstyle warnings in hadoop-auth package

2016-10-13 Thread Akira Ajisaka (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-13417:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   Status: Resolved  (was: Patch Available)

Committed this to trunk. Thanks [~lewuathe] for the contribution!

> Fix javac and checkstyle warnings in hadoop-auth package
> 
>
> Key: HADOOP-13417
> URL: https://issues.apache.org/jira/browse/HADOOP-13417
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13417.01.patch, HADOOP-13417.02.patch, 
> HADOOP-13417.03.patch
>
>
> Fix compile warnings generated after migrating JDK8.
> This is a sub-task of HADOOP-13369.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13686) Adding additional unit test for Trash (I)

2016-10-13 Thread Xiaoyu Yao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574302#comment-15574302
 ] 

Xiaoyu Yao commented on HADOOP-13686:
-

Thanks [~cheersyang] for the update. +1 for v7 patch and I will commit it 
shortly.

> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
>  Issue Type: Test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch, 
> HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch, 
> HADOOP-13686.06.patch, HADOOP-13686.07.patch
>
>
> This ticket is opened to track adding the forllowing unit test in 
> hadoop-common. 
> #test users can delete their own trash directory
> #test users can delete an empty directory and the directory is moved to trash
> #test fs.trash.interval with invalid values such as 0 or negative



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13723) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13723:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I have committed this to {{trunk}} branch. Thanks [~drankye] for review.

> AliyunOSSInputStream#read() should update read bytes stat correctly
> ---
>
> Key: HADOOP-13723
> URL: https://issues.apache.org/jira/browse/HADOOP-13723
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11007.000.patch
>
>
> {code}
>   @Override
>   public synchronized int read() throws IOException {
> ..
> if (statistics != null && byteRead >= 0) {
>   statistics.incrementBytesRead(1);
> }
> return byteRead;
>   }
> {code}
> I believe it should be {{statistics.incrementBytesRead(byteRead);}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13669) KMS Server should log exceptions before throwing

2016-10-13 Thread Xiao Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-13669:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed the addendum patch to the same branches. Thanks [~jojochuang] for the 
review and [~aw] for raising the issue.

> KMS Server should log exceptions before throwing
> 
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, 
> HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into 
> tomcat), it's not logged anywhere and we can only see the exception message 
> from client-side, but not the stacktrace. Logging the stacktrance would help 
> debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13061) Refactor erasure coders

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574121#comment-15574121
 ] 

Hadoop QA commented on HADOOP-13061:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 10 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
43s{color} | {color:green} root: The patch generated 0 new + 194 unchanged - 24 
fixed = 194 total (was 218) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m  
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 60m 
40s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13061 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833271/HADOOP-13061.18.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 46a642489451 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/testReport/ |
| modules | C: hadoop-common-project/hadoop-common 
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10782/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Refactor erasure coders
> ---
>
> Key: HADOOP-13061
> URL:

[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class

2016-10-13 Thread Rakesh R (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574080#comment-15574080
 ] 

Rakesh R commented on HADOOP-13715:
---

Thanks [~drankye] for the useful thoughts. I could see encryption feature has 
exposed APIs like, HdfsFileStatus has {{#getFileEncryptionInfo()}} exposed and 
FileStatus contains {{#isEncrypted()}} flag, to tell whether the underlying 
file or directory is encrypted or not. imho, EC could expose APIs in similar 
lines.

> Add isErasureCoded() API to FileStatus class
> 
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> Per the discussion in 
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
>  I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that 
> tools and downstream applications can tell if it needs to treat a file 
> differently.
> Hadoop tools that can benefit from this effort include: distcp and 
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574051#comment-15574051
 ] 

Hadoop QA commented on HADOOP-13032:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m  
4s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 13 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  5m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  2m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  9m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  6m 
58s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 58s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 52s{color} | {color:orange} root: The patch generated 79 new + 1559 
unchanged - 41 fixed = 1638 total (was 1600) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  6m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  3m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m  
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m  
9s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
16s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
55s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
34s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-openstack in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
35s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
24s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
36s{color} | {color:green}

[jira] [Commented] (HADOOP-13686) Adding additional unit test for Trash (I)

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574016#comment-15574016
 ] 

Hadoop QA commented on HADOOP-13686:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 76 unchanged - 1 fixed = 76 total (was 77) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13686 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833276/HADOOP-13686.07.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f621f4f2914b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10783/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10783/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
>  Issue Type: Test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch, 
> HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch, 
> HADOOP-13686.06.patch, HADOOP-13686.07.patch

[jira] [Commented] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573941#comment-15573941
 ] 

Yulei Li commented on HADOOP-13618:
---

I have sent an e-mail, please test the patch, thanks.

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
>

[jira] [Comment Edited] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573866#comment-15573866
 ] 

Yulei Li edited comment on HADOOP-13618 at 10/14/16 2:44 AM:
-

Both cases means the object contains spaces and %20 string. I will package the 
jar and send it to you, including HADOOP-13617 and HADOOP-13618.


was (Author: charlse):
Both cases means the object contains spaces and %20 string. I will package the 
jar and send it to you, including HADOOP-13617 and HADOOP-13618.

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
>

[jira] [Updated] (HADOOP-13686) Adding additional unit test for Trash (I)

2016-10-13 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated HADOOP-13686:
-
Attachment: HADOOP-13686.07.patch

V7 patch removed the throws IOException in AuditableTrashPolicy#initialize 
function according to the change of HADOOP-13700. No functional change. 

> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
>  Issue Type: Test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch, 
> HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch, 
> HADOOP-13686.06.patch, HADOOP-13686.07.patch
>
>
> This ticket is opened to track adding the forllowing unit test in 
> hadoop-common. 
> #test users can delete their own trash directory
> #test users can delete an empty directory and the directory is moved to trash
> #test fs.trash.interval with invalid values such as 0 or negative



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13535) Add jetty6 acceptor startup issue workaround to branch-2

2016-10-13 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573880#comment-15573880
 ] 

Brahma Reddy Battula commented on HADOOP-13535:
---

I think, we should backport to branch-2.7 also..?

> Add jetty6 acceptor startup issue workaround to branch-2
> 
>
> Key: HADOOP-13535
> URL: https://issues.apache.org/jira/browse/HADOOP-13535
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Wei-Chiu Chuang
>Assignee: Min Shen
> Fix For: 2.8.0
>
> Attachments: HADOOP-13535.001.patch, HADOOP-13535.002.patch, 
> HADOOP-13535.003.patch
>
>
> After HADOOP-12765 is committed to branch-2, the handling of SSL connection 
> by HttpServer2 may suffer the same Jetty bug described in HADOOP-10588. We 
> should consider adding the same workaround for SSL connection.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573866#comment-15573866
 ] 

Yulei Li edited comment on HADOOP-13618 at 10/14/16 2:12 AM:
-

Both cases means the object contains spaces and %20 string. I will package the 
jar and send it to you, including HADOOP-13617 and HADOOP-13618.


was (Author: charlse):
Both cases means the object contains spaces and %20 string. I will package the 
jar and send it to you, just wait.

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
>

[jira] [Commented] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573866#comment-15573866
 ] 

Yulei Li commented on HADOOP-13618:
---

Both cases means the object contains spaces and %20 string. I will package the 
jar and send it to you, just wait.

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
>

[jira] [Commented] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Steve Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573852#comment-15573852
 ] 

Steve Yang commented on HADOOP-13618:
-

Hi Yulei, 
Yes, please send me or provide a location where I can download it. Not sure 
what "both cases" mean here. Does it mean this patch also contains the fix for 
HADOOP-13617 as well?

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
>

[jira] [Commented] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573845#comment-15573845
 ] 

Hadoop QA commented on HADOOP-13618:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
11s{color} | {color:green} hadoop-openstack in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 31s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13618 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833270/HADOOP-13618.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 79c3c9bf221b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10781/testReport/ |
| modules | C: hadoop-tools/hadoop-openstack U: hadoop-tools/hadoop-openstack |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10781/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We

[jira] [Commented] (HADOOP-13506) Redundant groupid warning in child projects

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573835#comment-15573835
 ] 

Hadoop QA commented on HADOOP-13506:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 39m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 15m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 22m 
51s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 37m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 15m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  1m 
13s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 20m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hadoop-annotations in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
8s{color} | {color:green} hadoop-project-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
9s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
11s{color} | {color:green} hadoop-maven-plugins in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
19s{color} | {color:green} hadoop-minikdc in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
5s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
11s{color} | {color:green} hadoop-auth-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
22s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
7s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 14s{color} 
| {color:red} hadoop-common-project in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 52s{color} 
| {color:red}

[jira] [Updated] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13618:
--
Status: Patch Available  (was: Open)

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
>

[jira] [Work stopped] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-13618 stopped by Yulei Li.
-
> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
>

[jira] [Work started] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-13618 started by Yulei Li.
-
> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
>

[jira] [Updated] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13618:
--
Flags: Patch
Fix Version/s: 3.0.0-alpha2

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at 
>

[jira] [Updated] (HADOOP-13061) Refactor erasure coders

2016-10-13 Thread Kai Sasaki (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Sasaki updated HADOOP-13061:

Attachment: HADOOP-13061.18.patch

> Refactor erasure coders
> ---
>
> Key: HADOOP-13061
> URL: https://issues.apache.org/jira/browse/HADOOP-13061
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Kai Sasaki
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, 
> HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, 
> HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, 
> HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch, 
> HADOOP-13061.12.patch, HADOOP-13061.13.patch, HADOOP-13061.14.patch, 
> HADOOP-13061.15.patch, HADOOP-13061.16.patch, HADOOP-13061.17.patch, 
> HADOOP-13061.18.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13618) IllegalArgumentException when accessing Swift object with name containing space character

2016-10-13 Thread Yulei Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13618:
--
Attachment: HADOOP-13618.patch

I have tested both cases you mentioned using the patch, the error no more 
occured. You can test the patch in your envrioment, but you should first apply 
the patch in your source code, then build the openstack jar. If you can't do 
that, please tell me, I will e-mail the jar package for you.

> IllegalArgumentException when accessing Swift object with name containing 
> space character
> -
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar 
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle 
> Storage Service which is Swift-based:
> DataFrame df = 
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching 
> records.csv" where the object name "non-matching records.csv" contains a 
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem 
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13: 
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.(SwiftObjectPath.java:59)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at 
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at 
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the 
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing 
> SwiftNativeFileSystem against URI 
> swift://Linda.oracleswift/non-matching%20records.csv and working dir 
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem 
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration 
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the 
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space 
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching 
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack 
> Swift API, a different error is raised. This time somehow the path separator 
> '/' after the container name 'Linda' got encoded by 
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  HEAD 
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
>  => 400
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at 
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at 
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at

[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class

2016-10-13 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573793#comment-15573793
 ] 

Kai Zheng commented on HADOOP-13715:


bq. are we sure that a simple boolean is actually the value desired?
People may want to expose {{ErasureCodingPolicy getErasureCodingPolicy()}} 
other than the simple boolean value because it can contain more info and be 
flexible. But it may require that ErasureCodingPolicy concept be defined in 
hadoop level other than just for HDFS, however since most hadoop file system 
won't need the concept so it doesn't sound like elegant. So far as I can view, 
isErasureCoded is good to have as the existing isEncrypted. If applications 
need drill into the erasure coding internal, it will need to depend on the HDFS 
client API then.

It does be a good chance to add/change the API at this time, targeting the 3.0 
release.

> Add isErasureCoded() API to FileStatus class
> 
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> Per the discussion in 
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
>  I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that 
> tools and downstream applications can tell if it needs to treat a file 
> differently.
> Hadoop tools that can benefit from this effort include: distcp and 
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents

2016-10-13 Thread Ding Fei (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573749#comment-15573749
 ] 

Ding Fei commented on HADOOP-13708:
---

Uploaded "-4" patch is applicable in my latest local repository. 
Please try again! Sorry for messing up

> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, 
> HADOOP-13708-3.patch, HADOOP-13708-4.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents. 
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573741#comment-15573741
 ] 

Hadoop QA commented on HADOOP-13708:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  7s{color} 
| {color:red} HADOOP-13708 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13708 |
| GITHUB PR | https://github.com/apache/hadoop/pull/140 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10780/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, 
> HADOOP-13708-3.patch, HADOOP-13708-4.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents. 
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13708) Fix a few typos in site *.md documents

2016-10-13 Thread Ding Fei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ding Fei updated HADOOP-13708:
--
Attachment: HADOOP-13708-4.patch

> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, 
> HADOOP-13708-3.patch, HADOOP-13708-4.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents. 
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13061) Refactor erasure coders

2016-10-13 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573731#comment-15573731
 ] 

Kai Zheng commented on HADOOP-13061:


+1 once above two are addressed. [~andrew.wang] might also give a look? Thanks.

> Refactor erasure coders
> ---
>
> Key: HADOOP-13061
> URL: https://issues.apache.org/jira/browse/HADOOP-13061
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Kai Sasaki
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, 
> HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, 
> HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, 
> HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch, 
> HADOOP-13061.12.patch, HADOOP-13061.13.patch, HADOOP-13061.14.patch, 
> HADOOP-13061.15.patch, HADOOP-13061.16.patch, HADOOP-13061.17.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13723) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573729#comment-15573729
 ] 

Hadoop QA commented on HADOOP-13723:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
11s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13723 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833250/HDFS-11007.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 266fa7f25c8a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10779/testReport/ |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10779/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> AliyunOSSInputStream#read() should update read bytes stat correctly
> ---
>
> Key: HADOOP-13723
> URL: https://issues.apache.org/jira/browse/HADOOP-13723
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11007.000.patch
>
>
> {code}
>   @Override
>   public

[jira] [Commented] (HADOOP-13723) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573725#comment-15573725
 ] 

Mingliang Liu commented on HADOOP-13723:


Thanks for taking care of this!

> AliyunOSSInputStream#read() should update read bytes stat correctly
> ---
>
> Key: HADOOP-13723
> URL: https://issues.apache.org/jira/browse/HADOOP-13723
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11007.000.patch
>
>
> {code}
>   @Override
>   public synchronized int read() throws IOException {
> ..
> if (statistics != null && byteRead >= 0) {
>   statistics.incrementBytesRead(1);
> }
> return byteRead;
>   }
> {code}
> I believe it should be {{statistics.incrementBytesRead(byteRead);}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13061) Refactor erasure coders

2016-10-13 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573727#comment-15573727
 ] 

Kai Zheng commented on HADOOP-13061:


Another one is, we don't need to have this configuration because it's only for 
internal and very specific to raw coder implementation.
{code}
  public static final String IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_KEY =
  "io.erasurecode.codec.allow-change-inputs";
  public static final boolean IO_ERASURECODE_CODEC_ALLOW_CHANGE_INPUTS_DEFAULT
  = false;
{code}

> Refactor erasure coders
> ---
>
> Key: HADOOP-13061
> URL: https://issues.apache.org/jira/browse/HADOOP-13061
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Kai Sasaki
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, 
> HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, 
> HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, 
> HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch, 
> HADOOP-13061.12.patch, HADOOP-13061.13.patch, HADOOP-13061.14.patch, 
> HADOOP-13061.15.patch, HADOOP-13061.16.patch, HADOOP-13061.17.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13061) Refactor erasure coders

2016-10-13 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573713#comment-15573713
 ] 

Kai Zheng commented on HADOOP-13061:


Thanks [~lewuathe] for the update! Just find one issue, looks like the failure 
{{TestCommonConfigurationFields}} is related:
{noformat}
java.lang.AssertionError: core-default.xml has 1 properties missing in  class 
org.apache.hadoop.fs.CommonConfigurationKeys  class 
org.apache.hadoop.fs.CommonConfigurationKeysPublic  class 
org.apache.hadoop.fs.local.LocalConfigKeys  class 
org.apache.hadoop.fs.ftp.FtpConfigKeys  class 
org.apache.hadoop.ha.SshFenceByTcpPort  class 
org.apache.hadoop.security.LdapGroupsMapping  class 
org.apache.hadoop.ha.ZKFailoverController  class 
org.apache.hadoop.security.ssl.SSLFactory  class 
org.apache.hadoop.security.CompositeGroupsMapping  class 
org.apache.hadoop.io.erasurecode.rawcoder.CoderUtil

at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:563)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
{noformat}

> Refactor erasure coders
> ---
>
> Key: HADOOP-13061
> URL: https://issues.apache.org/jira/browse/HADOOP-13061
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Kai Sasaki
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, 
> HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, 
> HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, 
> HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch, 
> HADOOP-13061.12.patch, HADOOP-13061.13.patch, HADOOP-13061.14.patch, 
> HADOOP-13061.15.patch, HADOOP-13061.16.patch, HADOOP-13061.17.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents

2016-10-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573706#comment-15573706
 ] 

ASF GitHub Bot commented on HADOOP-13708:
-

Github user danix800 closed the pull request at:

https://github.com/apache/hadoop/pull/141


> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, 
> HADOOP-13708-3.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents. 
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13723) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573703#comment-15573703
 ] 

Hadoop QA commented on HADOOP-13723:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m  
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
12s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-11007 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833250/HDFS-11007.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux ba069d823a49 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17145/testReport/ |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/17145/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> AliyunOSSInputStream#read() should update read bytes stat correctly
> ---
>
> Key: HADOOP-13723
> URL: https://issues.apache.org/jira/browse/HADOOP-13723
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11007.000.patch
>
>
> {code}
>   @Override
>   public synchronized int read()

[jira] [Moved] (HADOOP-13723) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Kai Zheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng moved HDFS-11007 to HADOOP-13723:
---

Target Version/s: 3.0.0-alpha2  (was: 3.0.0-alpha2)
 Component/s: (was: tools)
  tools
 Key: HADOOP-13723  (was: HDFS-11007)
 Project: Hadoop Common  (was: Hadoop HDFS)

> AliyunOSSInputStream#read() should update read bytes stat correctly
> ---
>
> Key: HADOOP-13723
> URL: https://issues.apache.org/jira/browse/HADOOP-13723
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11007.000.patch
>
>
> {code}
>   @Override
>   public synchronized int read() throws IOException {
> ..
> if (statistics != null && byteRead >= 0) {
>   statistics.incrementBytesRead(1);
> }
> return byteRead;
>   }
> {code}
> I believe it should be {{statistics.incrementBytesRead(byteRead);}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-10225) Publish Maven javadoc and sources artifacts with Hadoop releases.

2016-10-13 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573673#comment-15573673
 ] 

Andrew Wang commented on HADOOP-10225:
--

Hi Lewis, thanks for working on this. As I said in an earlier comment, I was 
hoping to get this working without pulling in the release plugin. Since the 
current release script and instructions deploy everything except the javadoc 
jar, I think some smaller patch will also get the javadoc deployed.

> Publish Maven javadoc and sources artifacts with Hadoop releases.
> -
>
> Key: HADOOP-10225
> URL: https://issues.apache.org/jira/browse/HADOOP-10225
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build, documentation
>Affects Versions: 3.0.0-alpha1
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>  Labels: hadoop, javadoc, maven, sources
> Attachments: HADOOP-10225.patch
>
>
> Right now Maven javadoc and sources artifacts do not accompany Hadoop 
> releases within Maven central. This means that one needs to checkout source 
> code to DEBUG aspects of the codebase... this is not user friendly.
> The build script(s) should be amended to accommodate publication of javadoc 
> and sources artifacts alongside pom and jar artifacts. 
> Some history on this conversation can be seen below
> http://s.apache.org/7qR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573619#comment-15573619
 ] 

Hadoop QA commented on HADOOP-7352:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
5s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
53s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 31s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-7352 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833224/HADOOP-7352.005.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 3441bbb5a1e1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10776/testReport/ |
| modules | C: hadoop-common-project/hadoop-common 
hadoop-hdfs-project/hadoop-hdfs-client hadoop-tools/hadoop-distcp U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10776/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



>

[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.

2016-10-13 Thread Lei (Eddy) Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573616#comment-15573616
 ] 

Lei (Eddy) Xu commented on HADOOP-13449:


Thanks!

For the MS contract test, I am also playing that: I think that it should not 
verify the fields that S3AFileStatus does not support, i.e., {{atime}}.  We 
should also adjust contract test for that.

> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.wip.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13721) Remove stale method ViewFileSystem#getTrashCanLocation

2016-10-13 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573605#comment-15573605
 ] 

Andrew Wang commented on HADOOP-13721:
--

Good find Manoj. This is almost certainly supposed to be getTrashRoot and 
family.

Related question though, does the default getTrashRoot implementation work for 
ViewFileSystem? It seems like we need to defer to the resolved filesystem's 
implementation, since DFS for instance re-implements it to support encryption 
zones.

> Remove stale method ViewFileSystem#getTrashCanLocation
> --
>
> Key: HADOOP-13721
> URL: https://issues.apache.org/jira/browse/HADOOP-13721
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HADOOP-13721.01.patch
>
>
> {{ViewFileSystem}} which extends {{FileSystem}} has a public method 
> {{getTrashCanLocation}} which is neither overridden nor used by anybody. 
> Looks like it existed when the file was created, and also I see the 
> implementation returning homeDirectory which might not be the expected one in 
> cases of {{expunge}}. So, inclined to remove this stale and potentially 
> dangerous method unless anyone has any concerns. 
> {code}
>   public Path getTrashCanLocation(final Path f) throws FileNotFoundException {
> final InodeTree.ResolveResult res = 
>   fsState.resolve(getUriPath(f), true);
> return res.isInternalDir() ? null : 
> res.targetFileSystem.getHomeDirectory();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.

2016-10-13 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573607#comment-15573607
 ] 

Mingliang Liu commented on HADOOP-13449:


Thank you [~eddyxu] for your review! I'll address them in the v0 patch. I'm 
oncall this week so hopefully early next week I can post a working patch.

The unit test can not pass by now, as the UT assumes read (e.g. {{get}}) 
operations return a very valid {{S3AFileStatus}} which DynamoDBMetadataStaore 
doesn't provide yet. I have to read related patches in parallel to address this.

> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.wip.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13055) Implement linkMergeSlash for ViewFs

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573600#comment-15573600
 ] 

Hadoop QA commented on HADOOP-13055:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
40s{color} | {color:green} root: The patch generated 0 new + 148 unchanged - 13 
fixed = 148 total (was 161) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
48s{color} | {color:red} hadoop-common-project_hadoop-common generated 1 new + 
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 31s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 22s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13055 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833208/HADOOP-13055.03.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a837e217f7d6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javadoc | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10774/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10774/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results |

[jira] [Commented] (HADOOP-13659) Upgrade jaxb-api version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573597#comment-15573597
 ] 

Wei-Chiu Chuang commented on HADOOP-13659:
--

So this one is interesting. The development website of jaxb-api states the 
latest version is 2.2.11, which was released on October 14, 2014. There's no 
mention of 2.2.12. But maven repository does have 2.1.12 and its release date 
is October 20, 2014.

> Upgrade jaxb-api version
> 
>
> Key: HADOOP-13659
> URL: https://issues.apache.org/jira/browse/HADOOP-13659
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13659.001.patch
>
>
> We're currently pulling in version 2.2.2 - I think we should upgrade to the 
> latest 2.2.12.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13717) Shell scripts call hadoop_verify_logdir even when command is not started as daemon

2016-10-13 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573587#comment-15573587
 ] 

Andrew Wang commented on HADOOP-13717:
--

Besides the Mover, I'd also include the new intra-DN balancer in the collection 
of special commands. It's currently not daemon-enabled.

I don't have strong opinions here, but it seems like if someone is not 
specifying the "--daemon" flag, then they don't care about daemon things like 
pid files and log dirs for stdout/stderr. The audit log is an interesting case, 
but I think app-specific logging should be checked in the app, not the shell 
scripts (which are generic).

I think some combination of a) and b) is appropriate.

Regarding a), I don't think the balancer is commonly run in the background; 
Bigtop and CDH don't have balancer init scripts for instance. So can remove 
daemonization, also for Mover if it has it.

Regarding b), I'd prefer to short-circuit to hadoop_java_exec, but b) is 
alright too. I think there should be some generic fix for when "--daemon" isn't 
specified, because of user expectations.

Happy to try a patch if you agree.

> Shell scripts call hadoop_verify_logdir even when command is not started as 
> daemon
> --
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the 
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which 
> specifies the log location isn't even used here, since the command is being 
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into 
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13659) Upgrade jaxb-api version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-13659:
-
Target Version/s: 3.0.0-alpha2

> Upgrade jaxb-api version
> 
>
> Key: HADOOP-13659
> URL: https://issues.apache.org/jira/browse/HADOOP-13659
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13659.001.patch
>
>
> We're currently pulling in version 2.2.2 - I think we should upgrade to the 
> latest 2.2.12.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13660) Upgrade commons-configuration version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573581#comment-15573581
 ] 

Wei-Chiu Chuang commented on HADOOP-13660:
--

So I do have one extra comment: the last 1.x commons-configuration release, 
1.10, was released in 2013 October. Should we consider migrating to 2.1 where 
there seems to be more active development? My concern is that if we do not 
migrate now, we have to wait until next Hadoop major release, and by then it 
might be 7 years since the last 1.x release of commons-configuration.

However, given that this dependency is used lightly in Hadoop codebase and I am 
not aware of downstream applications relying on Hadoop's commons-configuration 
version, it may be fine if we have to migrate to commons-configuration 2.x in a 
Hadoop 3.x minor release.

> Upgrade commons-configuration version
> -
>
> Key: HADOOP-13660
> URL: https://issues.apache.org/jira/browse/HADOOP-13660
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13660.001.patch
>
>
> We're currently pulling in version 1.6 - I think we should upgrade to the 
> latest 1.10.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.

2016-10-13 Thread Lei (Eddy) Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573579#comment-15573579
 ] 

Lei (Eddy) Xu commented on HADOOP-13449:


Thanks for posting the patch,[~liuml07]. 

The concept looks very reasonable in general. And I like the schema. I 
understand that this is a WIP patch. So I'd list the following suggestions for 
reference. 

* Using local dynamodb local mode in test is really nice.
* We should store more metadata beside {{is_directory}} in metadata. so that we 
can reconstruct {{S3AFileStatus}} in {{itemToPathMetadata()}}.  Especially, do 
you think it is worth to store {{S3AFileStatus#isEmptyDirectory}} as well?
* To this extend, I think {{PathMetadata}} should take {{S3AFileStatus()}} 
instead of {{FileStatus}}. So that S3 specific attributes are more easily to be 
obtained. 
* {{DynamoDBMetadataStore}} should have a {{deleteTable}} on par with 
{{initTable}}. Both {{initTable}} and {{deleteTable}} should be package-wide 
visible so that it can be used from CLI tools. 
* 
{code}
try {
   table.waitForActive();
} catch (InterruptedException e) {
   LOG.warn("Interrupted while waiting for DynamoDB table {} active",
  tableName, e);
}
{code}

Should it throw {{IOE}} to indicate the failure?

* When do {{table.query()}} in {{listChildren()}},  the query might return 
partial results because the returned dataset is large. You can use {{ 
QueryResult#LastEvaluatedKey()}} for the following calls.

* DynamoDB {{tableName}} should be able to be specified in configuration, i.e., 
considering that multiple ETL jobs might running against the same dataset with 
different purposes and different lifetimes, using different tables could allow 
such jobs managed the lifetime of dynamodb tables by themself. 

Thanks for the nice work!

> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.wip.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics

2016-10-13 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HADOOP-13032:
---
Attachment: HADOOP-13032.002.patch

[~cmccabe] Can you kindly review this ([HADOOP-13435] code is included as this 
patch depends on it) if you find time? Thanks.

> Refactor FileSystem$Statistics to use StorageStatistics
> ---
>
> Key: HADOOP-13032
> URL: https://issues.apache.org/jira/browse/HADOOP-13032
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13032.000.patch, HADOOP-13032.001.patch, 
> HADOOP-13032.002.patch
>
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics. 
> This jira is to track the effort of moving the {{Statistics}} class out of 
> {{FileSystem}}, and make it use that new interface.
> We should keep the thread local implementation. Benefits are:
> # they could be used in both {{FileContext}} and {{FileSystem}}
> # unified stats data structure
> # shorter source code
> Please note this will be an backwards-incompatible change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13660) Upgrade commons-configuration version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573564#comment-15573564
 ] 

Wei-Chiu Chuang commented on HADOOP-13660:
--

I am aware that Whirr, Accumulo uses commons-configuration.

Accumulo specifies its version, 1.6.
Whirr specifies its version, 1.7.

Since both does not inherit version from Hadoop, it should be fine unless there 
are other projects that inherit commons-configuration version from Hadoop.

So I am +1 for this patch, and I'd like to commit this patch by end of this 
week. Please shout out if any one has any concerns.

> Upgrade commons-configuration version
> -
>
> Key: HADOOP-13660
> URL: https://issues.apache.org/jira/browse/HADOOP-13660
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13660.001.patch
>
>
> We're currently pulling in version 1.6 - I think we should upgrade to the 
> latest 1.10.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13660) Upgrade commons-configuration version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-13660:
-
Target Version/s: 3.0.0-alpha2

> Upgrade commons-configuration version
> -
>
> Key: HADOOP-13660
> URL: https://issues.apache.org/jira/browse/HADOOP-13660
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13660.001.patch
>
>
> We're currently pulling in version 1.6 - I think we should upgrade to the 
> latest 1.10.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-8928) Add ability to reset topologies on master nodes

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573471#comment-15573471
 ] 

Hadoop QA commented on HADOOP-8928:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} HADOOP-8928 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-8928 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12729963/HADOOP-8928.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10777/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add ability to reset topologies on master nodes
> ---
>
> Key: HADOOP-8928
> URL: https://issues.apache.org/jira/browse/HADOOP-8928
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net
>Affects Versions: 2.0.2-alpha, 3.0.0-alpha1
>Reporter: Shinichi Yamashita
>  Labels: BB2015-05-TBR
> Attachments: HADOOP-8928.patch, HADOOP-8928.txt
>
>
> For a topology decision of DataNode and TaskTracker, ScriptBasedMapping 
> (probably TableMapping) confirms HashMap first.
> To decide topology of DataNode and TaskTracker again, it is necessary to 
> restart NameNode and JobTracker.
> Therefore, it is necessary to change (or clear) HashMap function without 
> restarting NameNode and JobTracker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-8928) Add ability to reset topologies on master nodes

2016-10-13 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573466#comment-15573466
 ] 

Mingliang Liu commented on HADOOP-8928:
---

This seems a useful tool. Any updates? Thanks.

> Add ability to reset topologies on master nodes
> ---
>
> Key: HADOOP-8928
> URL: https://issues.apache.org/jira/browse/HADOOP-8928
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net
>Affects Versions: 2.0.2-alpha, 3.0.0-alpha1
>Reporter: Shinichi Yamashita
>  Labels: BB2015-05-TBR
> Attachments: HADOOP-8928.patch, HADOOP-8928.txt
>
>
> For a topology decision of DataNode and TaskTracker, ScriptBasedMapping 
> (probably TableMapping) confirms HashMap first.
> To decide topology of DataNode and TaskTracker again, it is necessary to 
> restart NameNode and JobTracker.
> Therefore, it is necessary to change (or clear) HashMap function without 
> restarting NameNode and JobTracker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13721) Remove stale method ViewFileSystem#getTrashCanLocation

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573456#comment-15573456
 ] 

Hadoop QA commented on HADOOP-13721:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 61 unchanged - 1 fixed = 61 total (was 62) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13721 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833212/HADOOP-13721.01.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a921af304247 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0a85d07 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10775/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10775/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Remove stale method ViewFileSystem#getTrashCanLocation
> --
>
> Key: HADOOP-13721
> URL: https://issues.apache.org/jira/browse/HADOOP-13721
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Manoj Govindassamy
>Assignee: Manoj

[jira] [Commented] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits

2016-10-13 Thread Eric Badger (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573445#comment-15573445
 ] 

Eric Badger commented on HADOOP-13709:
--

Thanks, [~jlowe], [~daryn]. I will work on a patch that implements a shutdown 
hook to destroy all of the created subprocesses. 

> Clean up subprocesses spawned by Shell.java:runCommand when the shell process 
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will 
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O 
> waiting for the return value of the subprocess that was spawned. We need to 
> allow for the subprocess to be interrupted and killed when the shell process 
> gets killed. Currently the JVM will shutdown and all of the subprocesses will 
> be orphaned and not killed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits

2016-10-13 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573435#comment-15573435
 ] 

Jason Lowe commented on HADOOP-13709:
-

Good catch, Daryn!  I missed the memory-model races introduced by moving the 
stdout processing to a new thread.  We could fix that with appropriate 
synchronization in the derived classes.  However Shell is explicitly marked 
Public, so it's prudent to avoid disrupting that too much.

It would be really nice to be able to kill the subprocess if the thread 
executing Shell is interrupted, but without moving the stdout processing to 
another thread I don't see a good way to do that.  Reading from a stream 
doesn't seem to be interruptible in practice.  If we can't make the Shell 
command reliably interruptible then the next best thing is to make sure we 
don't leave them lingering when this process exits so we can fix the problem 
described in YARN-5641.  Having a live Shell list that we can traverse on 
shutdown should fix that particular issue.



> Clean up subprocesses spawned by Shell.java:runCommand when the shell process 
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will 
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O 
> waiting for the return value of the subprocess that was spawned. We need to 
> allow for the subprocess to be interrupted and killed when the shell process 
> gets killed. Currently the JVM will shutdown and all of the subprocesses will 
> be orphaned and not killed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error

2016-10-13 Thread John Zhuge (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HADOOP-7352:
---
Attachment: HADOOP-7352.005.patch

Patch 005:
* Fix javadoc for {{FileSystem#listStatus}}

> FileSystem#listStatus should throw IOE upon access error
> 
>
> Key: HADOOP-7352
> URL: https://issues.apache.org/jira/browse/HADOOP-7352
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: Matt Foley
>Assignee: John Zhuge
> Attachments: HADOOP-7352.001.patch, HADOOP-7352.002.patch, 
> HADOOP-7352.003.patch, HADOOP-7352.004.patch, HADOOP-7352.005.patch
>
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should 
> throw FileNotFoundException instead of returning null, when the target 
> directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus 
> still may return null, when the target directory exists but does not grant 
> read permission.  This causes NPE in many callers, for all the reasons cited 
> in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for 
> examples.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573336#comment-15573336
 ] 

Hadoop QA commented on HADOOP-13560:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
39s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
41s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
15s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 44s{color} 
| {color:red} root-jdk1.8.0_101 with JDK v1.8.0_101 generated 1 new + 857 
unchanged - 0 fixed = 858 total (was 857) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 44s{color} 
| {color:red} root-jdk1.7.0_111 with JDK v1.7.0_111 generated 2 new + 949 
unchanged - 1 fixed = 951 total (was 950) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 29s{color} | {color:orange} root: The patch generated 26 new + 48 unchanged 
- 2 fixed = 74 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 62 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 35s{color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
28s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_111. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
|

[jira] [Updated] (HADOOP-13721) Remove stale method ViewFileSystem#getTrashCanLocation

2016-10-13 Thread Manoj Govindassamy (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HADOOP-13721:

Attachment: HADOOP-13721.01.patch

* {{expunge}} command relies on {{fs.trash.classname}} property or the default 
impl {{TrashPolicyDefault}}. The default Trash policy  relies on 
{{FileSystem#getTrashRoot}} to get the trash directory. 
* Currently I couldn't find any users for 
{{ViewFileSystem#getTrashCanLocation}}. Also the implementation is wrong, which 
return the home directory for the trash can.

So, inclined to remove this method {{ViewFileSystem#getTrashCanLocation}} for 
good. 

[~andrew.wang] can you please let me know if I am overlooking something very 
obvious here.

> Remove stale method ViewFileSystem#getTrashCanLocation
> --
>
> Key: HADOOP-13721
> URL: https://issues.apache.org/jira/browse/HADOOP-13721
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HADOOP-13721.01.patch
>
>
> {{ViewFileSystem}} which extends {{FileSystem}} has a public method 
> {{getTrashCanLocation}} which is neither overridden nor used by anybody. 
> Looks like it existed when the file was created, and also I see the 
> implementation returning homeDirectory which might not be the expected one in 
> cases of {{expunge}}. So, inclined to remove this stale and potentially 
> dangerous method unless anyone has any concerns. 
> {code}
>   public Path getTrashCanLocation(final Path f) throws FileNotFoundException {
> final InodeTree.ResolveResult res = 
>   fsState.resolve(getUriPath(f), true);
> return res.isInternalDir() ? null : 
> res.targetFileSystem.getHomeDirectory();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573331#comment-15573331
 ] 

Hadoop QA commented on HADOOP-13703:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
51s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
47s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
46s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 46s{color} 
| {color:red} root-jdk1.7.0_111 with JDK v1.7.0_111 generated 1 new + 949 
unchanged - 1 fixed = 950 total (was 950) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 30s{color} | {color:orange} root: The patch generated 20 new + 46 unchanged 
- 4 fixed = 66 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 62 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 44s{color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
27s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_111. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 34s{color} |

[jira] [Updated] (HADOOP-13721) Remove stale method ViewFileSystem#getTrashCanLocation

2016-10-13 Thread Manoj Govindassamy (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HADOOP-13721:

Status: Patch Available  (was: Open)

> Remove stale method ViewFileSystem#getTrashCanLocation
> --
>
> Key: HADOOP-13721
> URL: https://issues.apache.org/jira/browse/HADOOP-13721
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HADOOP-13721.01.patch
>
>
> {{ViewFileSystem}} which extends {{FileSystem}} has a public method 
> {{getTrashCanLocation}} which is neither overridden nor used by anybody. 
> Looks like it existed when the file was created, and also I see the 
> implementation returning homeDirectory which might not be the expected one in 
> cases of {{expunge}}. So, inclined to remove this stale and potentially 
> dangerous method unless anyone has any concerns. 
> {code}
>   public Path getTrashCanLocation(final Path f) throws FileNotFoundException {
> final InodeTree.ResolveResult res = 
>   fsState.resolve(getUriPath(f), true);
> return res.isInternalDir() ? null : 
> res.targetFileSystem.getHomeDirectory();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error

2016-10-13 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573335#comment-15573335
 ] 

Xiao Chen commented on HADOOP-7352:
---

Thanks John for the new patch. Sorry I missed this earlier: we should also 
update FileSystem.java's {{listStatus}} javadoc to say it no longer returns 
null. +1 after that though.

Will let this float until next Tuesday to let [~ste...@apache.org], [~daryn] 
and other watchers take a look.

> FileSystem#listStatus should throw IOE upon access error
> 
>
> Key: HADOOP-7352
> URL: https://issues.apache.org/jira/browse/HADOOP-7352
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: Matt Foley
>Assignee: John Zhuge
> Attachments: HADOOP-7352.001.patch, HADOOP-7352.002.patch, 
> HADOOP-7352.003.patch, HADOOP-7352.004.patch
>
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should 
> throw FileNotFoundException instead of returning null, when the target 
> directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus 
> still may return null, when the target directory exists but does not grant 
> read permission.  This causes NPE in many callers, for all the reasons cited 
> in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for 
> examples.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13722) Code cleanup -- ViewFileSystem and InodeTree

2016-10-13 Thread Manoj Govindassamy (JIRA)

Manoj Govindassamy created HADOOP-13722:
---

 Summary: Code cleanup -- ViewFileSystem and InodeTree
 Key: HADOOP-13722
 URL: https://issues.apache.org/jira/browse/HADOOP-13722
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
Priority: Minor


ViewFileSystem is the FileSystem for viewfs:// and its uses InodeTree to manage 
the mount points. These files being very old, don't quit adhere to the styling 
and coding standards. Will do code cleanup of these files as part of this jira. 
No new functionalities or tests will be added as part of this jira. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits

2016-10-13 Thread Eric Badger (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573286#comment-15573286
 ] 

Eric Badger commented on HADOOP-13709:
--

[~daryn], good catch! I was able to recreate the hung test locally and took a 
jstack. Looks like it is indeed deadlocking between threads reading the stdout 
stream. 

I'm not super thrilled about using a shutdown hook to clean up the processes, 
but in this case I think it will work if we can't think of anything else. 
[~jlowe], do you have any additional insights? 

{noformat}
Found one Java-level deadlock:
=
"pool-4-thread-1":
  waiting to lock monitor 0x7f592d78 (object 0xd67243a0, a 
java.lang.UNIXProcess$ProcessPipeInputStream),
  which is held by "Thread-0"
"Thread-0":
  waiting to lock monitor 0x7f58f8006008 (object 0xd672ec28, a 
java.io.InputStreamReader),
  which is held by "pool-4-thread-1"

Java stack information for the threads listed above:
===
"pool-4-thread-1":
at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
- waiting to lock <0xd67243a0> (a 
java.lang.UNIXProcess$ProcessPipeInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0xd672ec28> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.read1(BufferedReader.java:212)
at java.io.BufferedReader.read(BufferedReader.java:286)
- locked <0xd672ec28> (a java.io.InputStreamReader)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:1214)
at org.apache.hadoop.util.Shell$2.call(Shell.java:965)
at org.apache.hadoop.util.Shell$2.call(Shell.java:962)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Thread-0":
at java.io.BufferedReader.close(BufferedReader.java:522)
- waiting to lock <0xd672ec28> (a java.io.InputStreamReader)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1029)
- locked <0xd67243a0> (a 
java.lang.UNIXProcess$ProcessPipeInputStream)
at org.apache.hadoop.util.Shell.run(Shell.java:883)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1201)
at 
org.apache.hadoop.util.TestShell.testShellCommandTimerLeak(TestShell.java:241)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Found 1 deadlock.
{noformat}

> Clean up subprocesses spawned by Shell.java:runCommand when the shell process 
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will 
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O 
> waiting for the return value of the subprocess that was spawned. We need to 
> allow for the subprocess to be interrupted and killed when the shell process 
> gets killed. Currently the JVM will shutdown and all of the subprocesses will 
> be orphaned and not killed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail:

[jira] [Created] (HADOOP-13721) Remove stale method ViewFileSystem#getTrashCanLocation

2016-10-13 Thread Manoj Govindassamy (JIRA)

Manoj Govindassamy created HADOOP-13721:
---

 Summary: Remove stale method ViewFileSystem#getTrashCanLocation
 Key: HADOOP-13721
 URL: https://issues.apache.org/jira/browse/HADOOP-13721
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
Priority: Minor



{{ViewFileSystem}} which extends {{FileSystem}} has a public method 
{{getTrashCanLocation}} which is neither overridden nor used by anybody. Looks 
like it existed when the file was created, and also I see the implementation 
returning homeDirectory which might not be the expected one in cases of 
{{expunge}}. So, inclined to remove this stale and potentially dangerous method 
unless anyone has any concerns. 

{code}
  public Path getTrashCanLocation(final Path f) throws FileNotFoundException {
final InodeTree.ResolveResult res = 
  fsState.resolve(getUriPath(f), true);
return res.isInternalDir() ? null : res.targetFileSystem.getHomeDirectory();
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13720) Add more info to "token ... is expired" message

2016-10-13 Thread Yongjun Zhang (JIRA)

Yongjun Zhang created HADOOP-13720:
--

 Summary: Add more info to "token ... is expired" message
 Key: HADOOP-13720
 URL: https://issues.apache.org/jira/browse/HADOOP-13720
 Project: Hadoop Common
  Issue Type: Bug
  Components: common, security
Reporter: Yongjun Zhang


Currently AbstractDelegationTokenSecretM anager$checkToken does

{code}
  protected DelegationTokenInformation checkToken(TokenIdent identifier)
  throws InvalidToken {
assert Thread.holdsLock(this);
DelegationTokenInformation info = getTokenInfo(identifier);
if (info == null) {
  throw new InvalidToken("token (" + identifier.toString()
  + ") can't be found in cache");
}
if (info.getRenewDate() < Time.now()) {
  throw new InvalidToken("token (" + identifier.toString() + ") is 
expired");
}
return info;
  } 
{code}

When a token is expried, we throw the above exception without printing out the 
{{info.getRenewDate()}} in the message. If we print it out, we could know for 
how long the token has not been renewed. This will help us investigate certain 
issues.

Create this jira as a request to add that part.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13055) Implement linkMergeSlash for ViewFs

2016-10-13 Thread Manoj Govindassamy (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HADOOP-13055:

Attachment: HADOOP-13055.03.patch

Attached v03 patch, an enhancement on top of [~zhz]'s path. It addresses the 
following

1. More verification for valid linkMergeSlash mount table entries
2. Mountable list update even for linkMergeSlash entry so that callers of 
FileSystem.getChildFileSystems() gets to see the slash mountpoint
3. Unit test which makes use of Federated topology and multiple mount tables. 
and some basic verification. Will add more tests around renames, distcp in the 
follow up jiras.
4. Little bit of code cleanups in {{InodeTree}} around the areas where new 
fixes are added. More extensive code cleanup in {{ViewFileSystem}} and 
{{InodeTree}} will be done as part of a separate jira.

[~andrew.wang], [~eddyxu], [~zhz], can you please take a look at the patch and 
let me know your comments ?

> Implement linkMergeSlash for ViewFs
> ---
>
> Key: HADOOP-13055
> URL: https://issues.apache.org/jira/browse/HADOOP-13055
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs, viewfs
>Reporter: Zhe Zhang
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-13055.00.patch, HADOOP-13055.01.patch, 
> HADOOP-13055.02.patch, HADOOP-13055.03.patch
>
>
> In a multi-cluster environment it is sometimes useful to operate on the root 
> / slash directory of an HDFS cluster. E.g., list all top level directories. 
> Quoting the comment in {{ViewFs}}:
> {code}
>  *   A special case of the merge mount is where mount table's root is merged
>  *   with the root (slash) of another file system:
>  *   
>  *   fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/
>  *   
>  *   In this cases the root of the mount table is merged with the root of
>  *hdfs://nn99/  
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13661) Upgrade HTrace version

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573235#comment-15573235
 ] 

Wei-Chiu Chuang commented on HADOOP-13661:
--

Downstream projects that I know of use htrace:
HBase, Accumulo

Hbase explicitly defines its own htrace version 3.1.0-incubating, and even if I 
switch to hadoop-3.0 profile it sticks to htrace 3.1.0-incubating.

Accumulo does the same.

So since it doesn't break these two downstream projects I am +1 for the 02 
patch. I'll wait until end of Friday so that anyone has a chance to comment.

> Upgrade HTrace version
> --
>
> Key: HADOOP-13661
> URL: https://issues.apache.org/jira/browse/HADOOP-13661
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-13661.001.patch, HADOOP-13661.002.patch
>
>
> We're currently pulling in version 4.0.1-incubating - I think we should 
> upgrade to the latest 4.1.0-incubating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13708) Fix a few typos in site *.md documents

2016-10-13 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573202#comment-15573202
 ] 

Andrew Wang commented on HADOOP-13708:
--

Hi [~danix800], could you attach a single consolidated patch? The "-3" patch 
looks like a partial. I'm not that familiar with the github PR model either.

> Fix a few typos in site *.md documents
> --
>
> Key: HADOOP-13708
> URL: https://issues.apache.org/jira/browse/HADOOP-13708
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.0
>Reporter: Ding Fei
>Assignee: Ding Fei
>Priority: Minor
> Attachments: HADOOP-13708-1.patch, HADOOP-13708-2.patch, 
> HADOOP-13708-3.patch, HADOOP-13708.patch
>
>
> Fix several typos in site *.md documents. 
> Touched documents listed:
> * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/testing.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/notation.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
> * hadoop-common-project/hadoop-common/src/site/markdown/filesystem/model.md
> * 
> hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md
> * hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
> * hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Mavin Martin (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573196#comment-15573196
 ] 

Mavin Martin commented on HADOOP-13024:
---

Hi Hudson, Looks like you merged patch 9.  Can you merge patch 10 instead?  It 
has some checkstyle fixes.  
https://issues.apache.org/jira/secure/attachment/12833179/HADOOP-13024.patch.10

> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.10, HADOOP-13024.patch.3, HADOOP-13024.patch.4, 
> HADOOP-13024.patch.5, HADOOP-13024.patch.6, HADOOP-13024.patch.7, 
> HADOOP-13024.patch.8, HADOOP-13024.patch.9
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573184#comment-15573184
 ] 

Hadoop QA commented on HADOOP-12082:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 42s{color} | {color:orange} root: The patch generated 124 new + 155 
unchanged - 2 fixed = 279 total (was 157) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
22s{color} | {color:red} hadoop-auth in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
43s{color} | {color:red} hadoop-common-project/hadoop-auth generated 3 new + 0 
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
19s{color} | {color:red} hadoop-auth in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
38s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-auth |
|  |  Found reliance on default encoding in 
org.apache.hadoop.security.authentication.server.LdapAuthenticationHandler.authenticate(HttpServletRequest,
 HttpServletResponse):in 
org.apache.hadoop.security.authentication.server.LdapAuthenticationHandler.authenticate(HttpServletRequest,
 HttpServletResponse): new String(byte[])  At

[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing

2016-10-13 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573141#comment-15573141
 ] 

Xiao Chen commented on HADOOP-13669:


Thanks [~jojochuang]! I'll commit end of today then.

+1 on Yetus should improve on this.. I browsed through Yetus code but didn't 
find an easy way to fix this. Will create a jira.

> KMS Server should log exceptions before throwing
> 
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, 
> HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into 
> tomcat), it's not logged anywhere and we can only see the exception message 
> from client-side, but not the stacktrace. Logging the stacktrance would help 
> debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573121#comment-15573121
 ] 

Hudson commented on HADOOP-13024:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10607 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10607/])
HADOOP-13024. Distcp with -delete feature on raw data not implemented. (jing9: 
rev 0a85d079838f532a13ca237300386d1b3bc1b178)
* (edit) 
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
* (edit) 
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpWithRawXAttrs.java
* (edit) 
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/DistCpTestUtils.java
* (edit) 
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java


> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.10, HADOOP-13024.patch.3, HADOOP-13024.patch.4, 
> HADOOP-13024.patch.5, HADOOP-13024.patch.6, HADOOP-13024.patch.7, 
> HADOOP-13024.patch.8, HADOOP-13024.patch.9
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter

2016-10-13 Thread Hrishikesh Gadre (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573112#comment-15573112
 ] 

Hrishikesh Gadre commented on HADOOP-12082:
---

[~benoyantony] Done !

> Support multiple authentication schemes via AuthenticationFilter
> 
>
> Key: HADOOP-12082
> URL: https://issues.apache.org/jira/browse/HADOOP-12082
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: HADOOP-12082-001.patch, HADOOP-12082-002.patch, 
> HADOOP-12082.patch, hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, 
> hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, 
> hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, 
> multi-scheme-auth-support-poc.patch
>
>
> The requirement is to support LDAP based authentication scheme via Hadoop 
> AuthenticationFilter. HADOOP-9054 added a support to plug-in custom 
> authentication scheme (in addition to Kerberos) via 
> AltKerberosAuthenticationHandler class. But it is based on selecting the 
> authentication mechanism based on User-Agent HTTP header which does not 
> conform to HTTP protocol semantics.
> As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html]
> - HTTP protocol provides a simple challenge-response authentication mechanism 
> that can be used by a server to challenge a client request and by a client to 
> provide the necessary authentication information. 
> - This mechanism is initiated by server sending the 401 (Authenticate) 
> response with ‘WWW-Authenticate’ header which includes at least one challenge 
> that indicates the authentication scheme(s) and parameters applicable to the 
> Request-URI. 
> - In case server supports multiple authentication schemes, it may return 
> multiple challenges with a 401 (Authenticate) response, and each challenge 
> may use a different auth-scheme. 
> - A user agent MUST choose to use the strongest auth-scheme it understands 
> and request credentials from the user based upon that challenge.
> The existing Hadoop authentication filter implementation supports Kerberos 
> authentication scheme and uses ‘Negotiate’ as the challenge as part of 
> ‘WWW-Authenticate’ response header. As per the following documentation, 
> ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows 
> NTLM) authentication schemes.
> [SPNEGO-based Kerberos and NTLM HTTP 
> Authentication|http://tools.ietf.org/html/rfc4559]
> [Understanding HTTP 
> Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx]
> On the other hand for LDAP authentication, typically ‘Basic’ authentication 
> scheme is used (Note TLS is mandatory with Basic authentication scheme).
> http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html
> Hence for this feature, the idea would be to provide a custom implementation 
> of Hadoop AuthenticationHandler and Authenticator interfaces which would 
> support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via 
> Basic auth challenge). During the authentication phase, it would send both 
> the challenges and let client pick the appropriate one. If client responds 
> with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos 
> authentication. If client responds with an ‘Authorization’ header tagged with 
> ‘Basic’ - it will use LDAP authentication.
> Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be 
> configured to use one scheme over the other e.g.
> - curl tool supports option to use either Kerberos (via --negotiate flag) or 
> username/password based authentication (via --basic and -u flags). 
> - Apache HttpClient library can be configured to use specific authentication 
> scheme.
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
> Typically web browsers automatically choose an authentication scheme based on 
> a notion of “strength” of security. e.g. take a look at the [design of Chrome 
> browser for HTTP 
> authentication|https://www.chromium.org/developers/design-documents/http-authentication]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Jing Zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HADOOP-13024:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

I've committed the patch to trunk, branch-2 and 2.8. Thanks for the 
contribution, [~mavinmar...@gmail.com]!

> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.10, HADOOP-13024.patch.3, HADOOP-13024.patch.4, 
> HADOOP-13024.patch.5, HADOOP-13024.patch.6, HADOOP-13024.patch.7, 
> HADOOP-13024.patch.8, HADOOP-13024.patch.9
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12774) s3a should use UGI.getCurrentUser.getShortname() for username

2016-10-13 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573070#comment-15573070
 ] 

Steve Loughran commented on HADOOP-12774:
-

last patch in PR addresses the issues; Yetus somehow missed it.

I'd like to hold off for HADOOP1-13560; IMO that one is ready for commit, and 
we'll have to see what problems arise downstream. FWIW I have been testing 
spark with it, not seen problems

> s3a should use UGI.getCurrentUser.getShortname() for username
> -
>
> Key: HADOOP-12774
> URL: https://issues.apache.org/jira/browse/HADOOP-12774
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3a users {{System.getProperty("user.name")}} to get the username for the 
> homedir. This is wrong, as it doesn't work on a YARN app where the identity 
> is set by HADOOP_USER_NAME, or in a doAs clause.
> Obviously, {{UGI.getCurrentUser.getShortname()}} provides that name, 
> everywhere. 
> This is a simple change in the source, though testing is harder ... probably 
> best to try in a doAs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits

2016-10-13 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573049#comment-15573049
 ] 

Daryn Sharp commented on HADOOP-13709:
--

If you catch it in the act, I bet its going to be some kind of deadlock between 
the jvm's shell reaper thread, the main process, and the new process doing the 
parseExecResult.  There's too many threads accessing highly synchronized 
readers/streams.

I'm uneasy about this patch in general.  Changing the semantics of anything in 
hadoop-common is fraught with peril.  Subclasses are not expecting their 
parseExecResult to be run in a thread which may introduce subtle errors 
including but not limited to synchronization/volatility issues.  Or if method 
uses thread locals but now it's in a different thread, etc.

While interrupting a thread in a shell command would be nice, I think the more 
general requirement for yarn is that processes are not left running.  It's 
probably a safer assumption that no hadoop service wants lingering processes.  
Perhaps create a set of all running shell instances.  A shutdown hook could 
call destroy() on all the instances still registered.

> Clean up subprocesses spawned by Shell.java:runCommand when the shell process 
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will 
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O 
> waiting for the return value of the subprocess that was spawned. We need to 
> allow for the subprocess to be interrupted and killed when the shell process 
> gets killed. Currently the JVM will shutdown and all of the subprocesses will 
> be orphaned and not killed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13716) Add LambdaTestUtils class for tests to make use of

2016-10-13 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573030#comment-15573030
 ] 

Steve Loughran commented on HADOOP-13716:
-

I couldn't think of good names either, as the linear one is still less than any 
exponential backoff strategy.



> Add LambdaTestUtils class for tests to make use of
> --
>
> Key: HADOOP-13716
> URL: https://issues.apache.org/jira/browse/HADOOP-13716
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: test
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13716-001.patch, HADOOP-13716-002.patch, 
> HADOOP-13716-003.patch
>
>
> To make our tests robust against timing problems and eventual consistent 
> stores, we need to do more spin & wait for state.
> We have some code in {{GenericTestUtils.waitFor}} to await a condition being 
> met, but the predicate it calls doesn't throw exceptions, there's no way for 
> a probe to throw an exception, and all you get is the eventual "timed out" 
> message. 
> We can do better, and in closure-ready languages (scala & scalatest, groovy 
> and some slider code) we've examples to follow. Some of that work has been 
> reimplemented slightly in {{S3ATestUtils.eventually}}
> I propose adding a class in the test tree, {{Eventually}} to be a 
> successor/replacement for these.
> # has an eventually/waitfor operation taking a predicate that throws an 
> exception
> # has an "evaluate" exception which tries to evaluate an answer until the 
> operation stops raising an exception. (again, from scalatest)
> # plugin backoff strategies (from Scalatest; lets you do exponential as well 
> as linear)
> # option of adding a special handler to generate the failure exception (e.g. 
> run more detailed diagnostics for the exception text, etc).
> # be Java 8 lambda expression friendly
> # be testable and tested itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13669) KMS Server should log exceptions before throwing

2016-10-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573022#comment-15573022
 ] 

Wei-Chiu Chuang commented on HADOOP-13669:
--

Got it [~xiaochen] 
The findbugs warning refers to trunk code, not the patch.
{quote}
cd /testptch/hadoop/hadoop-common-project/hadoop-kms
mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-0 -Ptest-patch 
-DskipTests test-compile findbugs:findbugs -DskipTests=true > 
/testptch/hadoop/patchprocess/branch-findbugs-hadoop-common-project_hadoop-kms.txt
 2>&1
Elapsed:   0m 23s

hadoop-common-project/hadoop-kms in trunk has 2 extant Findbugs warnings.
{quote}

So Yetus should not flag findbugs warning in trunk. Probably worth an 
improvement.

+1 from me. I'll leave it untouched until EOB in case Allen has additional 
comments.

> KMS Server should log exceptions before throwing
> 
>
> Key: HADOOP-13669
> URL: https://issues.apache.org/jira/browse/HADOOP-13669
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: kms
>Reporter: Xiao Chen
>Assignee: Suraj Acharya
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13369.2.patch, HADOOP-13369.patch, 
> HADOOP-13369.patch.1, HADOOP-13669.addendum.patch, trigger.patch
>
>
> In some recent investigation, it turns out when KMS throws an exception (into 
> tomcat), it's not logged anywhere and we can only see the exception message 
> from client-side, but not the stacktrace. Logging the stacktrance would help 
> debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter

2016-10-13 Thread Hrishikesh Gadre (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrishikesh Gadre updated HADOOP-12082:
--
Status: Patch Available  (was: Open)

> Support multiple authentication schemes via AuthenticationFilter
> 
>
> Key: HADOOP-12082
> URL: https://issues.apache.org/jira/browse/HADOOP-12082
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: HADOOP-12082-001.patch, HADOOP-12082-002.patch, 
> HADOOP-12082.patch, hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, 
> hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, 
> hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, 
> multi-scheme-auth-support-poc.patch
>
>
> The requirement is to support LDAP based authentication scheme via Hadoop 
> AuthenticationFilter. HADOOP-9054 added a support to plug-in custom 
> authentication scheme (in addition to Kerberos) via 
> AltKerberosAuthenticationHandler class. But it is based on selecting the 
> authentication mechanism based on User-Agent HTTP header which does not 
> conform to HTTP protocol semantics.
> As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html]
> - HTTP protocol provides a simple challenge-response authentication mechanism 
> that can be used by a server to challenge a client request and by a client to 
> provide the necessary authentication information. 
> - This mechanism is initiated by server sending the 401 (Authenticate) 
> response with ‘WWW-Authenticate’ header which includes at least one challenge 
> that indicates the authentication scheme(s) and parameters applicable to the 
> Request-URI. 
> - In case server supports multiple authentication schemes, it may return 
> multiple challenges with a 401 (Authenticate) response, and each challenge 
> may use a different auth-scheme. 
> - A user agent MUST choose to use the strongest auth-scheme it understands 
> and request credentials from the user based upon that challenge.
> The existing Hadoop authentication filter implementation supports Kerberos 
> authentication scheme and uses ‘Negotiate’ as the challenge as part of 
> ‘WWW-Authenticate’ response header. As per the following documentation, 
> ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows 
> NTLM) authentication schemes.
> [SPNEGO-based Kerberos and NTLM HTTP 
> Authentication|http://tools.ietf.org/html/rfc4559]
> [Understanding HTTP 
> Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx]
> On the other hand for LDAP authentication, typically ‘Basic’ authentication 
> scheme is used (Note TLS is mandatory with Basic authentication scheme).
> http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html
> Hence for this feature, the idea would be to provide a custom implementation 
> of Hadoop AuthenticationHandler and Authenticator interfaces which would 
> support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via 
> Basic auth challenge). During the authentication phase, it would send both 
> the challenges and let client pick the appropriate one. If client responds 
> with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos 
> authentication. If client responds with an ‘Authorization’ header tagged with 
> ‘Basic’ - it will use LDAP authentication.
> Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be 
> configured to use one scheme over the other e.g.
> - curl tool supports option to use either Kerberos (via --negotiate flag) or 
> username/password based authentication (via --basic and -u flags). 
> - Apache HttpClient library can be configured to use specific authentication 
> scheme.
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
> Typically web browsers automatically choose an authentication scheme based on 
> a notion of “strength” of security. e.g. take a look at the [design of Chrome 
> browser for HTTP 
> authentication|https://www.chromium.org/developers/design-documents/http-authentication]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13486) Method invocation in log can be replaced by variable because the variable's toString method contain more info

2016-10-13 Thread Nemo Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572949#comment-15572949
 ] 

Nemo Chen commented on HADOOP-13486:


I think in order to uniquely identify a client connection, we need both the 
host address and the port number. Plus, the replacement with the toString 
method makes the code more readable. Please let me know if this makes sense.

> Method invocation in log can be replaced by variable because the variable's 
> toString method contain more info 
> --
>
> Key: HADOOP-13486
> URL: https://issues.apache.org/jira/browse/HADOOP-13486
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Nemo Chen
>  Labels: easyfix, easytest
>
> Similar to the fix in HADOOP-6419, in file:
> hadoop-rel-release-2.7.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> {code}
> Connection c = (Connection)key.attachment();
> ...
> LOG.info(Thread.currentThread().getName() + ": readAndProcess from client " + 
> c.getHostAddress() + " threw exception [" + e + "]", (e instanceof 
> WrappedRpcServerException) ? null : e);
> ...
> {code}
> in class Connection, the toString method contains both getHostAddress() and 
> remotePort
> {code}
> public String toString() {
>   return getHostAddress() + ":" + remotePort; 
> }
> {code}
> Therefore the c.getHostAddress() should be replaced by c for simplicity and 
> information wise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace

2016-10-13 Thread Hanisha Koneru (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572931#comment-15572931
 ] 

Hanisha Koneru commented on HADOOP-13710:
-

Thank you [~jojochuang] and [~arpitagarwal] for reviewing and committing the 
patch.

> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-13710.000.patch
>
>
> CachingGetSpaceUsed thread is typically interrupted when the node is 
> shutdown. Since this is a routine operation, there seems not much value to 
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN  fs.CachingGetSpaceUsed 
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh 
> disk information
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13037) Azure Data Lake Client: Support Azure data lake as a file system in Hadoop

2016-10-13 Thread Vishwajeet Dusane (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572919#comment-15572919
 ] 

Vishwajeet Dusane commented on HADOOP-13037:


Thanks [~chris.douglas], i think i generated patch against trunk from feature 
branch hence it pulled all the changeset from trunk. I will regenerate and 
submit patch against trunk only.

> Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
> --
>
> Key: HADOOP-13037
> URL: https://issues.apache.org/jira/browse/HADOOP-13037
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/azure, tools
>Reporter: Shrikant Naidu
>Assignee: Vishwajeet Dusane
> Fix For: 2.9.0
>
> Attachments: HADOOP-13037 Proposal.pdf, HADOOP-13037-001.patch
>
>
> The jira proposes an improvement over HADOOP-12666 to remove webhdfs 
> dependencies from the ADL file system client and build out a standalone 
> client. At a high level, this approach would extend the Hadoop file system 
> class to provide an implementation for accessing Azure Data Lake. The scheme 
> used for accessing the file system will continue to be 
> adl://.azuredatalake.net/path/to/file. 
> The Azure Data Lake Cloud Store will continue to provide a webHDFS rest 
> interface. The client will  access the ADLS store using WebHDFS Rest APIs 
> provided by the ADLS store. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13703) S3ABlockOutputStream to pass Yetus & Jenkins

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572915#comment-15572915
 ] 

Hadoop QA commented on HADOOP-13703:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 24 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
36s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
56s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
15s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  7m 10s{color} 
| {color:red} root-jdk1.7.0_111 with JDK v1.7.0_111 generated 1 new + 949 
unchanged - 1 fixed = 950 total (was 950) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 37s{color} | {color:orange} root: The patch generated 20 new + 46 unchanged 
- 4 fixed = 66 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 62 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
54s{color} | {color:green} hadoop-common in the patch passed with JDK 
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
29s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_111. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m

[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572927#comment-15572927
 ] 

Hadoop QA commented on HADOOP-13024:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
3s{color} | {color:blue} The patch file was not named according to hadoop's 
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute 
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 12s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated 
2 new + 69 unchanged - 4 fixed = 71 total (was 73) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m  
7s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13024 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833179/HADOOP-13024.patch.10 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 947f10182699 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 332a61f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10770/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10770/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10770/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Distcp with -delete feature on raw data not implemented
> ---
>
> Key:

[jira] [Commented] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter

2016-10-13 Thread Benoy Antony (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572920#comment-15572920
 ] 

Benoy Antony commented on HADOOP-12082:
---

Looks good. Could you please "submit patch"  ?

> Support multiple authentication schemes via AuthenticationFilter
> 
>
> Key: HADOOP-12082
> URL: https://issues.apache.org/jira/browse/HADOOP-12082
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: HADOOP-12082-001.patch, HADOOP-12082-002.patch, 
> HADOOP-12082.patch, hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, 
> hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, 
> hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, 
> multi-scheme-auth-support-poc.patch
>
>
> The requirement is to support LDAP based authentication scheme via Hadoop 
> AuthenticationFilter. HADOOP-9054 added a support to plug-in custom 
> authentication scheme (in addition to Kerberos) via 
> AltKerberosAuthenticationHandler class. But it is based on selecting the 
> authentication mechanism based on User-Agent HTTP header which does not 
> conform to HTTP protocol semantics.
> As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html]
> - HTTP protocol provides a simple challenge-response authentication mechanism 
> that can be used by a server to challenge a client request and by a client to 
> provide the necessary authentication information. 
> - This mechanism is initiated by server sending the 401 (Authenticate) 
> response with ‘WWW-Authenticate’ header which includes at least one challenge 
> that indicates the authentication scheme(s) and parameters applicable to the 
> Request-URI. 
> - In case server supports multiple authentication schemes, it may return 
> multiple challenges with a 401 (Authenticate) response, and each challenge 
> may use a different auth-scheme. 
> - A user agent MUST choose to use the strongest auth-scheme it understands 
> and request credentials from the user based upon that challenge.
> The existing Hadoop authentication filter implementation supports Kerberos 
> authentication scheme and uses ‘Negotiate’ as the challenge as part of 
> ‘WWW-Authenticate’ response header. As per the following documentation, 
> ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows 
> NTLM) authentication schemes.
> [SPNEGO-based Kerberos and NTLM HTTP 
> Authentication|http://tools.ietf.org/html/rfc4559]
> [Understanding HTTP 
> Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx]
> On the other hand for LDAP authentication, typically ‘Basic’ authentication 
> scheme is used (Note TLS is mandatory with Basic authentication scheme).
> http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html
> Hence for this feature, the idea would be to provide a custom implementation 
> of Hadoop AuthenticationHandler and Authenticator interfaces which would 
> support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via 
> Basic auth challenge). During the authentication phase, it would send both 
> the challenges and let client pick the appropriate one. If client responds 
> with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos 
> authentication. If client responds with an ‘Authorization’ header tagged with 
> ‘Basic’ - it will use LDAP authentication.
> Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be 
> configured to use one scheme over the other e.g.
> - curl tool supports option to use either Kerberos (via --negotiate flag) or 
> username/password based authentication (via --basic and -u flags). 
> - Apache HttpClient library can be configured to use specific authentication 
> scheme.
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
> Typically web browsers automatically choose an authentication scheme based on 
> a notion of “strength” of security. e.g. take a look at the [design of Chrome 
> browser for HTTP 
> authentication|https://www.chromium.org/developers/design-documents/http-authentication]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13686) Adding additional unit test for Trash (I)

2016-10-13 Thread Xiaoyu Yao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572901#comment-15572901
 ] 

Xiaoyu Yao commented on HADOOP-13686:
-

[~cheersyang], can you rebase the patch v06 with the latest trunk changes from 
HADOOP-13700? Otherwise, looks good to me. 

> Adding additional unit test for Trash (I)
> -
>
> Key: HADOOP-13686
> URL: https://issues.apache.org/jira/browse/HADOOP-13686
> Project: Hadoop Common
>  Issue Type: Test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
> Attachments: HADOOP-13686.01.patch, HADOOP-13686.02.patch, 
> HADOOP-13686.03.patch, HADOOP-13686.04.patch, HADOOP-13686.05.patch, 
> HADOOP-13686.06.patch
>
>
> This ticket is opened to track adding the forllowing unit test in 
> hadoop-common. 
> #test users can delete their own trash directory
> #test users can delete an empty directory and the directory is moved to trash
> #test fs.trash.interval with invalid values such as 0 or negative



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13719) Update ASF logo in the web site

2016-10-13 Thread Chris Douglas (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572879#comment-15572879
 ] 

Chris Douglas commented on HADOOP-13719:


+1

bq. IIRC this was fixed and reverted recently as the new logo broke the site 
layout.
That was a different issue. The new logo from HADOOP-13184 was taller than the 
old one, and that broke the site layout. As long as the new ASF logo is the 
same height we shouldn't have a problem.

This should probably also replace the ASF feather logo we have on some pages.

> Update ASF logo in the web site
> ---
>
> Key: HADOOP-13719
> URL: https://issues.apache.org/jira/browse/HADOOP-13719
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: site
>Reporter: Akira Ajisaka
>
> The banner at the right side of the document is the previous version of asf 
> logo.
> !http://www.apache.org/images/asf_logo_wide.png!
> This issue is to replace the old logo with the new one.
> !https://www.apache.org/foundation/press/kit/asf_logo_wide.svg!
> https://www.apache.org/foundation/press/kit/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Mavin Martin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mavin Martin updated HADOOP-13024:
--
Attachment: HADOOP-13024.patch.10

Fixed checkstyle.

> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.10, HADOOP-13024.patch.3, HADOOP-13024.patch.4, 
> HADOOP-13024.patch.5, HADOOP-13024.patch.6, HADOOP-13024.patch.7, 
> HADOOP-13024.patch.8, HADOOP-13024.patch.9
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace

2016-10-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572859#comment-15572859
 ] 

Hudson commented on HADOOP-13710:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10605 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10605/])
HADOOP-13710. Supress CachingGetSpaceUsed from logging interrupted (arp: rev 
008122b3c927767ac96dc876124bc591e10c9df4)
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java


> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-13710.000.patch
>
>
> CachingGetSpaceUsed thread is typically interrupted when the node is 
> shutdown. Since this is a routine operation, there seems not much value to 
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN  fs.CachingGetSpaceUsed 
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh 
> disk information
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572855#comment-15572855
 ] 

Hadoop QA commented on HADOOP-13024:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue}  0m  
1s{color} | {color:blue} The patch file was not named according to hadoop's 
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute 
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 11s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated 
3 new + 71 unchanged - 2 fixed = 74 total (was 73) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 
10s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13024 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833173/HADOOP-13024.patch.9 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 9828da27bc0f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 9097e2e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10769/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10769/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10769/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024

[jira] [Commented] (HADOOP-13719) Update ASF logo in the web site

2016-10-13 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572819#comment-15572819
 ] 

Arpit Agarwal commented on HADOOP-13719:


Hi [~ajisakaa] IIRC this was fixed and reverted recently as the new logo broke 
the site layout.

> Update ASF logo in the web site
> ---
>
> Key: HADOOP-13719
> URL: https://issues.apache.org/jira/browse/HADOOP-13719
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: site
>Reporter: Akira Ajisaka
>
> The banner at the right side of the document is the previous version of asf 
> logo.
> !http://www.apache.org/images/asf_logo_wide.png!
> This issue is to replace the old logo with the new one.
> !https://www.apache.org/foundation/press/kit/asf_logo_wide.svg!
> https://www.apache.org/foundation/press/kit/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace

2016-10-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HADOOP-13710:
---
   Resolution: Fixed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

+1

Committed for 2.8.0. Thanks for the contribution [~hanishakoneru].

> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-13710.000.patch
>
>
> CachingGetSpaceUsed thread is typically interrupted when the node is 
> shutdown. Since this is a routine operation, there seems not much value to 
> print the stacktrace of an {{InterruptedException}}.
> {quote}
> 2016-10-11 10:02:25,894 WARN  fs.CachingGetSpaceUsed 
> (CachingGetSpaceUsed.java:run(180)) - Thread Interrupted waiting to refresh 
> disk information
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:176)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13710) Supress CachingGetSpaceUsed from logging interrupted exception stacktrace

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572791#comment-15572791
 ] 

Hadoop QA commented on HADOOP-13710:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13710 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832991/HDFS-13710.000.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 5a678003b39e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / b371c56 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10767/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10767/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Supress CachingGetSpaceUsed from logging interrupted exception stacktrace
> -
>
> Key: HADOOP-13710
> URL: https://issues.apache.org/jira/browse/HADOOP-13710
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Hanisha Koneru
>Priority:

[jira] [Updated] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Mavin Martin (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mavin Martin updated HADOOP-13024:
--
Attachment: HADOOP-13024.patch.9

Uploaded .9 with final changes.

> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.3, HADOOP-13024.patch.4, HADOOP-13024.patch.5, 
> HADOOP-13024.patch.6, HADOOP-13024.patch.7, HADOOP-13024.patch.8, 
> HADOOP-13024.patch.9
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13037) Azure Data Lake Client: Support Azure data lake as a file system in Hadoop

2016-10-13 Thread Chris Douglas (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572772#comment-15572772
 ] 

Chris Douglas commented on HADOOP-13037:


It looks like v001 accidentally reverts HDFS-10789.

> Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
> --
>
> Key: HADOOP-13037
> URL: https://issues.apache.org/jira/browse/HADOOP-13037
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/azure, tools
>Reporter: Shrikant Naidu
>Assignee: Vishwajeet Dusane
> Fix For: 2.9.0
>
> Attachments: HADOOP-13037 Proposal.pdf, HADOOP-13037-001.patch
>
>
> The jira proposes an improvement over HADOOP-12666 to remove webhdfs 
> dependencies from the ADL file system client and build out a standalone 
> client. At a high level, this approach would extend the Hadoop file system 
> class to provide an implementation for accessing Azure Data Lake. The scheme 
> used for accessing the file system will continue to be 
> adl://.azuredatalake.net/path/to/file. 
> The Azure Data Lake Cloud Store will continue to provide a webHDFS rest 
> interface. The client will  access the ADLS store using WebHDFS Rest APIs 
> provided by the ADLS store. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13565) KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request

2016-10-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572768#comment-15572768
 ] 

Hudson commented on HADOOP-13565:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10604 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10604/])
HADOOP-13565. KerberosAuthenticationHandler#authenticate should not (xyao: rev 
9097e2efe4c92d83c8fab88dc11be84505a6cab5)
* (edit) 
hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java


> KerberosAuthenticationHandler#authenticate should not rebuild SPN based on 
> client request
> -
>
> Key: HADOOP-13565
> URL: https://issues.apache.org/jira/browse/HADOOP-13565
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.5.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Fix For: 2.8.0
>
> Attachments: HADOOP-13565.00.patch
>
>
> In KerberosAuthenticationHandler#authenticate, we use canonicalized server 
> name derived from HTTP request to build server SPN and authenticate client. 
> This can be problematic if the HTTP client/server are running from a 
> non-local Kerberos realm that the local realm has trust with (e.g., NN UI).
> For example, 
> The server is running its HTTP endpoint using SPN from the client realm:
> hadoop.http.authentication.kerberos.principal
> HTTP/_HOST/TEST.COM
> When client sends request to namenode at http://NN1.example.com:50070 from 
> client.test@test.com.
> The client talks to KDC first and gets a service ticket 
> HTTP/NN1.example.com/TEST.COM to authenticate with the server via SPNEGO 
> negotiation. 
> The authentication will end up with either no valid credential error or 
> checksum failure depending on the HTTP client naming resolution or HTTP Host 
> field from the request header provided by the browser. 
> The root cause is KerberosUtil.getServicePrincipal("HTTP", serverName)}} will 
> always return a SPN with local realm (HTTP/nn.example@example.com) no 
> matter the server login SPN is from that domain or not. 
> The proposed fix is to change to use default server login principal (by 
> passing null as the 1st parameter to gssManager.createCredential()) instead. 
> This way we avoid dependency on HTTP client behavior (Host header or name 
> resolution like CNAME) or assumption on the local realm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13037) Azure Data Lake Client: Support Azure data lake as a file system in Hadoop

2016-10-13 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572762#comment-15572762
 ] 

Hadoop QA commented on HADOOP-13037:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} HADOOP-13037 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13037 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833170/HADOOP-13037-001.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10768/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
> --
>
> Key: HADOOP-13037
> URL: https://issues.apache.org/jira/browse/HADOOP-13037
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/azure, tools
>Reporter: Shrikant Naidu
>Assignee: Vishwajeet Dusane
> Fix For: 2.9.0
>
> Attachments: HADOOP-13037 Proposal.pdf, HADOOP-13037-001.patch
>
>
> The jira proposes an improvement over HADOOP-12666 to remove webhdfs 
> dependencies from the ADL file system client and build out a standalone 
> client. At a high level, this approach would extend the Hadoop file system 
> class to provide an implementation for accessing Azure Data Lake. The scheme 
> used for accessing the file system will continue to be 
> adl://.azuredatalake.net/path/to/file. 
> The Azure Data Lake Cloud Store will continue to provide a webHDFS rest 
> interface. The client will  access the ADLS store using WebHDFS Rest APIs 
> provided by the ADLS store. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13024) Distcp with -delete feature on raw data not implemented

2016-10-13 Thread Jing Zhao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572757#comment-15572757
 ] 

Jing Zhao commented on HADOOP-13024:


.8 patch looks pretty good to me. The test code refactoring looks great. Only 
one nitpicky comment:
# All the constant paths can be defined in DistCpConstants.java as following:
{code}
  public static final String NONE_PATH_NAME = "/NONE";
  public static final Path NONE_PATH = new Path(NONE_PATH_NAME);
  public static final Path RAW_NONE_PATH = new Path(
  HDFS_RESERVED_RAW_DIRECTORY_NAME + NONE_PATH_NAME);
{code}

+1 after addressing the comment.

> Distcp with -delete feature on raw data not implemented
> ---
>
> Key: HADOOP-13024
> URL: https://issues.apache.org/jira/browse/HADOOP-13024
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
> Attachments: HADOOP-13024.patch, HADOOP-13024.patch, 
> HADOOP-13024.patch.3, HADOOP-13024.patch.4, HADOOP-13024.patch.5, 
> HADOOP-13024.patch.6, HADOOP-13024.patch.7, HADOOP-13024.patch.8
>
>
> When doing distcp of raw data using -delete feature, following bug appears.
> {code}
> [root@xxx bin]# hadoop distcp -delete -update /.reserved/raw/tmp/a 
> /.reserved/raw/tmp/b
> 16/04/14 02:54:01 ERROR tools.DistCp: Exception encountered
> java.io.IOException: DistCp failure: Job job_xxx has failed: Job commit 
> failed: org.apache.hadoop.tools.CopyListing$InvalidInputException: The source 
> path 'hdfs://nn/.reserved/raw/tmp/b' starts with /.reserved/raw but the 
> target path 'hdfs://nn/NONE' does not. Either all or none of the paths must 
> have this prefix.
> at 
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:141)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at 
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at 
> org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.deleteMissing(CopyCommitter.java:244)
> at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:94)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
> at 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:187)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {code}
> The issue is not with the distributed copy, the issue is when it tries to 
> delete things in the target that no longer exist in the source, it 
> revalidates to make sure NONE is in the /.reserved/raw domain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13257) Improve Azure Data Lake contract tests.

2016-10-13 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-13257:
---
Issue Type: Sub-task  (was: Improvement)
Parent: HADOOP-13037

> Improve Azure Data Lake contract tests.
> ---
>
> Key: HADOOP-13257
> URL: https://issues.apache.org/jira/browse/HADOOP-13257
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Chris Nauroth
>Assignee: Vishwajeet Dusane
>
> HADOOP-12875 provided the initial implementation of the FileSystem contract 
> tests covering Azure Data Lake.  This issue tracks subsequent improvements on 
> those test suites for improved coverage and matching the specified semantics 
> more closely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13037) Azure Data Lake Client: Support Azure data lake as a file system in Hadoop

2016-10-13 Thread Vishwajeet Dusane (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishwajeet Dusane updated HADOOP-13037:
---
Status: Patch Available  (was: Open)

Trigger Hadoop QA build on the patch since the test-patch script is failing to 
build hadoop-common code. However compilation succeed without test-patch 
script. I will look into test-patch issue.

> Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
> --
>
> Key: HADOOP-13037
> URL: https://issues.apache.org/jira/browse/HADOOP-13037
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs, fs/azure, tools
>Reporter: Shrikant Naidu
>Assignee: Vishwajeet Dusane
> Fix For: 2.9.0
>
> Attachments: HADOOP-13037 Proposal.pdf, HADOOP-13037-001.patch
>
>
> The jira proposes an improvement over HADOOP-12666 to remove webhdfs 
> dependencies from the ADL file system client and build out a standalone 
> client. At a high level, this approach would extend the Hadoop file system 
> class to provide an implementation for accessing Azure Data Lake. The scheme 
> used for accessing the file system will continue to be 
> adl://.azuredatalake.net/path/to/file. 
> The Azure Data Lake Cloud Store will continue to provide a webHDFS rest 
> interface. The client will  access the ADLS store using WebHDFS Rest APIs 
> provided by the ADLS store. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12082) Support multiple authentication schemes via AuthenticationFilter

2016-10-13 Thread Hrishikesh Gadre (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrishikesh Gadre updated HADOOP-12082:
--
Attachment: HADOOP-12082-002.patch

[~benoyantony] Thanks for the review! Here is an updated patch which addresses 
the review comment. Please take a look.

> Support multiple authentication schemes via AuthenticationFilter
> 
>
> Key: HADOOP-12082
> URL: https://issues.apache.org/jira/browse/HADOOP-12082
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: HADOOP-12082-001.patch, HADOOP-12082-002.patch, 
> HADOOP-12082.patch, hadoop-ldap-auth-v2.patch, hadoop-ldap-auth-v3.patch, 
> hadoop-ldap-auth-v4.patch, hadoop-ldap-auth-v5.patch, 
> hadoop-ldap-auth-v6.patch, hadoop-ldap.patch, 
> multi-scheme-auth-support-poc.patch
>
>
> The requirement is to support LDAP based authentication scheme via Hadoop 
> AuthenticationFilter. HADOOP-9054 added a support to plug-in custom 
> authentication scheme (in addition to Kerberos) via 
> AltKerberosAuthenticationHandler class. But it is based on selecting the 
> authentication mechanism based on User-Agent HTTP header which does not 
> conform to HTTP protocol semantics.
> As per [RFC-2616|http://www.w3.org/Protocols/rfc2616/rfc2616.html]
> - HTTP protocol provides a simple challenge-response authentication mechanism 
> that can be used by a server to challenge a client request and by a client to 
> provide the necessary authentication information. 
> - This mechanism is initiated by server sending the 401 (Authenticate) 
> response with ‘WWW-Authenticate’ header which includes at least one challenge 
> that indicates the authentication scheme(s) and parameters applicable to the 
> Request-URI. 
> - In case server supports multiple authentication schemes, it may return 
> multiple challenges with a 401 (Authenticate) response, and each challenge 
> may use a different auth-scheme. 
> - A user agent MUST choose to use the strongest auth-scheme it understands 
> and request credentials from the user based upon that challenge.
> The existing Hadoop authentication filter implementation supports Kerberos 
> authentication scheme and uses ‘Negotiate’ as the challenge as part of 
> ‘WWW-Authenticate’ response header. As per the following documentation, 
> ‘Negotiate’ challenge scheme is only applicable to Kerberos (and Windows 
> NTLM) authentication schemes.
> [SPNEGO-based Kerberos and NTLM HTTP 
> Authentication|http://tools.ietf.org/html/rfc4559]
> [Understanding HTTP 
> Authentication|https://msdn.microsoft.com/en-us/library/ms789031%28v=vs.110%29.aspx]
> On the other hand for LDAP authentication, typically ‘Basic’ authentication 
> scheme is used (Note TLS is mandatory with Basic authentication scheme).
> http://httpd.apache.org/docs/trunk/mod/mod_authnz_ldap.html
> Hence for this feature, the idea would be to provide a custom implementation 
> of Hadoop AuthenticationHandler and Authenticator interfaces which would 
> support both schemes - Kerberos (via Negotiate auth challenge) and LDAP (via 
> Basic auth challenge). During the authentication phase, it would send both 
> the challenges and let client pick the appropriate one. If client responds 
> with an ‘Authorization’ header tagged with ‘Negotiate’ - it will use Kerberos 
> authentication. If client responds with an ‘Authorization’ header tagged with 
> ‘Basic’ - it will use LDAP authentication.
> Note - some HTTP clients (e.g. curl or Apache Http Java client) need to be 
> configured to use one scheme over the other e.g.
> - curl tool supports option to use either Kerberos (via --negotiate flag) or 
> username/password based authentication (via --basic and -u flags). 
> - Apache HttpClient library can be configured to use specific authentication 
> scheme.
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
> Typically web browsers automatically choose an authentication scheme based on 
> a notion of “strength” of security. e.g. take a look at the [design of Chrome 
> browser for HTTP 
> authentication|https://www.chromium.org/developers/design-documents/http-authentication]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

1 2 >

1 - 100 of 155 matches

Mail list logo