date:20160706

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Siddharth Seth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365652#comment-15365652
 ] 

Siddharth Seth commented on HADOOP-13335:
-

bq. There have been a ton of removals all over the place in trunk already. S3, 
hftp, ... lots and lots of places. Bear in mind that the last opportunity 
Hadoop had to remove content was over five years ago.
Good to know.

bq. Once again: they do not. I'm not sure how many more and different ways I 
can state this point. The rest of that discussion is pretty much moot since the 
first opportunity to make them function the same will be 4.x, easily years off.
And once again - the only difference is the additional YARN_* environment 
variables added on. You seem to agree that having these additional variables is 
unnecessary and confusing, and they should eventually be removed (maybe in 
4.x). If the intent is to deprecate the (yarn jar) command - I don't see the 
point of pointing users towards this when they invoke hadoop jar.

Another way of looking at this: if someone has set the YARN_* options - they 
would do this intentionally, and would be aware of the yarn jar command. If 
they use hadoop jar after this point for something else - they know exactly 
what they're doing, and hadoop should not try pointing them back at yarn jar.

bq. yarn-config.sh and hadoop-config.sh in a way that should be mostly 
conflict-free
Right there is another problem. "Mostly conflict-free" is not a great API, and 
adds more confusion.

bq. Deprecation warnings have always been added to the output throughout the 
history of Hadoop. Expect a lot more to show up in trunk.
Is this particular case supposed to be a deprecation warning ?
Most system will also have a mechanism to suppress deprecation messages; we 
should add such an option to the scripts in trunk.

Is there some place that the public API for the hadoop scripts is documented ? 
The only place I have seen references along with documentation is within the 
scripts itself, and in 'usage' output.



> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt

2016-07-06 Thread Xiao Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365629#comment-15365629
 ] 

Xiao Chen commented on HADOOP-12893:


Thanks Andrew for reviewing.

So Jetty, Xerces and snappy are all under ASFv2, and [the apache 
guide|http://www.apache.org/dev/licensing-howto.html#alv2-dep] says:
{{If the dependency supplies a NOTICE file, its contents must be analyzed and 
the relevant portions bubbled up into the top-level NOTICE file.}}
I'm unclear what exactly are the relevant portions either. :( Shall we file 
LEGAL jiras for each of them? Then we'll only include what legal considers 
relevant... (unless someone in the audience here has the expertise / past 
experience).

> Verify LICENSE.txt and NOTICE.txt
> -
>
> Key: HADOOP-12893
> URL: https://issues.apache.org/jira/browse/HADOOP-12893
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Xiao Chen
>Priority: Blocker
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, 
> HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, 
> HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, 
> HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, 
> HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, 
> HADOOP-12893.addendum-trunk.01.patch, HADOOP-12893.branch-2.01.patch, 
> HADOOP-12893.branch-2.6.01.patch, HADOOP-12893.branch-2.7.01.patch, 
> HADOOP-12893.branch-2.7.02.patch, HADOOP-12893.branch-2.7.3.01.patch
>
>
> We have many bundled dependencies in both the source and the binary artifacts 
> that are not in LICENSE.txt and NOTICE.txt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13348) delete spurious 'master' branch

2016-07-06 Thread Sean Busbey (JIRA)

Sean Busbey created HADOOP-13348:


 Summary: delete spurious 'master' branch
 Key: HADOOP-13348
 URL: https://issues.apache.org/jira/browse/HADOOP-13348
 Project: Hadoop Common
  Issue Type: Task
  Components: build
Reporter: Sean Busbey


Right now the git repo has a branch named 'master' in addition to our 'trunk' 
branch. Since 'master' is the common-place name of the 'most recent' branch in 
git repositories, this is misleading to new folks.

It looks like the branch is from ~11 months ago. We should remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Jun Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Gong updated HADOOP-13347:
--
Resolution: Duplicate
Status: Resolved  (was: Patch Available)

> Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to 
> .gitignore
> --
>
> Key: HADOOP-13347
> URL: https://issues.apache.org/jira/browse/HADOOP-13347
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Jun Gong
>Priority: Minor
> Attachments: HADOOP-13347.01.patch
>
>
> After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
> in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
> status}}' looks like the following:
> {code}
> $ git status
> On branch trunk
> Your branch is up-to-date with 'origin/trunk'.
> Untracked files:
>   (use "git add ..." to include in what will be committed)
>   hadoop-build-tools/src/main/resources/META-INF/
> nothing added to commit but untracked files present (use "git add" to track)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Jun Gong (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365464#comment-15365464
 ] 

Jun Gong commented on HADOOP-13347:
---

Thanks [~liuml07]! Close it now.

> Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to 
> .gitignore
> --
>
> Key: HADOOP-13347
> URL: https://issues.apache.org/jira/browse/HADOOP-13347
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Jun Gong
>Priority: Minor
> Attachments: HADOOP-13347.01.patch
>
>
> After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
> in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
> status}}' looks like the following:
> {code}
> $ git status
> On branch trunk
> Your branch is up-to-date with 'origin/trunk'.
> Untracked files:
>   (use "git add ..." to include in what will be committed)
>   hadoop-build-tools/src/main/resources/META-INF/
> nothing added to commit but untracked files present (use "git add" to track)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13329) Dockerfile doesn't work on Linux/ppc

2016-07-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365463#comment-15365463
 ] 

Hadoop QA commented on HADOOP-13329:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  9m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  8m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 13 new + 75 unchanged - 0 fixed = 
88 total (was 75) {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}  
0m  8s{color} | {color:orange} The patch generated 8 new + 124 unchanged - 0 
fixed = 132 total (was 124) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
11s{color} | {color:green} root in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
28s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12816088/HADOOP-13329.patch |
| JIRA Issue | HADOOP-13329 |
| Optional Tests |  asflicense  shellcheck  shelldocs  mvnsite  unit  |
| uname | Linux 9b15f7f5ffef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a3f93be |
| shellcheck | v0.4.4 |
| shellcheck | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/artifact/patchprocess/diff-patch-shellcheck.txt
 |
| shelldocs | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/artifact/patchprocess/diff-patch-shelldocs.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/artifact/patchprocess/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/testReport/ |
| asflicense | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/artifact/patchprocess/patch-asflicense-problems.txt
 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9934/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Dockerfile doesn't work on Linux/ppc
> 
>
> Key: HADOOP-13329
> URL: https://issues.apache.org/jira/browse/HADOOP-13329
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Amir Sanjar
> Attachments: HADOOP-13329.patch
>
>
> We need to rework how the Dockerfile is built to support both Linux/x86 and 
> Linux/PowerPC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Mingliang Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365462#comment-15365462
 ] 

Mingliang Liu commented on HADOOP-13347:


Thanks for reporting this. This is duplicate to [HADOOP-13298]. As [~busbey] 
pointed out, 
{quote}
that only masks the problem in git. It doesn't impact that maven tooling will 
presume something is amiss, since the build is not supposed to alter the source 
directories.
{quote}

> Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to 
> .gitignore
> --
>
> Key: HADOOP-13347
> URL: https://issues.apache.org/jira/browse/HADOOP-13347
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Jun Gong
>Priority: Minor
> Attachments: HADOOP-13347.01.patch
>
>
> After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
> in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
> status}}' looks like the following:
> {code}
> $ git status
> On branch trunk
> Your branch is up-to-date with 'origin/trunk'.
> Untracked files:
>   (use "git add ..." to include in what will be committed)
>   hadoop-build-tools/src/main/resources/META-INF/
> nothing added to commit but untracked files present (use "git add" to track)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Jun Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Gong updated HADOOP-13347:
--
Attachment: HADOOP-13347.01.patch

> Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to 
> .gitignore
> --
>
> Key: HADOOP-13347
> URL: https://issues.apache.org/jira/browse/HADOOP-13347
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Jun Gong
>Priority: Minor
> Attachments: HADOOP-13347.01.patch
>
>
> After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
> in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
> status}}' looks like the following:
> {code}
> $ git status
> On branch trunk
> Your branch is up-to-date with 'origin/trunk'.
> Untracked files:
>   (use "git add ..." to include in what will be committed)
>   hadoop-build-tools/src/main/resources/META-INF/
> nothing added to commit but untracked files present (use "git add" to track)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Jun Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Gong updated HADOOP-13347:
--
Status: Patch Available  (was: Open)

> Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to 
> .gitignore
> --
>
> Key: HADOOP-13347
> URL: https://issues.apache.org/jira/browse/HADOOP-13347
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Jun Gong
>Priority: Minor
> Attachments: HADOOP-13347.01.patch
>
>
> After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
> in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
> status}}' looks like the following:
> {code}
> $ git status
> On branch trunk
> Your branch is up-to-date with 'origin/trunk'.
> Untracked files:
>   (use "git add ..." to include in what will be committed)
>   hadoop-build-tools/src/main/resources/META-INF/
> nothing added to commit but untracked files present (use "git add" to track)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13347) Add META-INF(LICENSE.txt and NOTICE.txt) generated in hadoop-build-tools to .gitignore

2016-07-06 Thread Jun Gong (JIRA)

Jun Gong created HADOOP-13347:
-

 Summary: Add META-INF(LICENSE.txt and NOTICE.txt) generated in 
hadoop-build-tools to .gitignore
 Key: HADOOP-13347
 URL: https://issues.apache.org/jira/browse/HADOOP-13347
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Jun Gong
Priority: Minor


After running mvn package, there will be LICENSE.txt and NOTICE.txt generated 
in hadoop-build-tools/src/main/resources/META-INF/. Then the result of '{{git 
status}}' looks like the following:
{code}
$ git status
On branch trunk
Your branch is up-to-date with 'origin/trunk'.
Untracked files:
  (use "git add ..." to include in what will be committed)

hadoop-build-tools/src/main/resources/META-INF/

nothing added to commit but untracked files present (use "git add" to track)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer

2016-07-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365442#comment-15365442
 ] 

Hadoop QA commented on HADOOP-13346:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 8 new + 76 unchanged - 0 fixed = 84 total (was 76) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m  0s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 15s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12816540/HADOOP-13346.patch |
| JIRA Issue | HADOOP-13346 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 8a9be34ddf2c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8a9d293 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9933/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9933/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9933/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9933/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> DelegationTokenAuthenticationHandler writes via closed writer
> -
>
>

[jira] [Commented] (HADOOP-13329) Dockerfile doesn't work on Linux/ppc

2016-07-06 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365437#comment-15365437
 ] 

Allen Wittenauer commented on HADOOP-13329:
---

I think what we should probably do is actually make separate platform 
directories underneath the docker dir then modify start-build-env.sh to point 
to the directory that matches the platform.  This would mean the current 
Dockerfile would get moved to x86_64 or whatever.

Some specific comments:

* The protobuf integration makes me very nervous.  That doesn't look like a 
long lasting URL at all.  Isn't there a repo we can point to?
* Why the custom code for leveldb?
* We should probably rework the start-build-env.sh docker call to avoid having 
two copies of the hadoop-env-checks script.  We could have it copy the correct 
Dockerfile and the hadoop-env-checks script to some place under target or 
patchprocess and build it from there.
* The Apache Yetus marker is missing which means this will completely blow up 
when used against the Apache build infrastructure

... speaking of, I can't really test this out until that machine gets re-hooked 
into the infra. :(


> Dockerfile doesn't work on Linux/ppc
> 
>
> Key: HADOOP-13329
> URL: https://issues.apache.org/jira/browse/HADOOP-13329
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Amir Sanjar
> Attachments: HADOOP-13329.patch
>
>
> We need to rework how the Dockerfile is built to support both Linux/x86 and 
> Linux/PowerPC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-9159) FileSystem class should override .equals(Object)

2016-07-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365433#comment-15365433
 ] 

Hadoop QA commented on HADOOP-9159:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  9m 12s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
|   | hadoop.fs.TestFilterFileSystem |
|   | hadoop.fs.TestHarFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12561804/HADOOP-9159.patch |
| JIRA Issue | HADOOP-9159 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux b4022201b035 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8a9d293 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9932/artifact/patchprocess/whitespace-eol.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9932/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9932/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9932/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FileSystem class should override .equals(Object)
> 
>
> Key: HADOOP-9159
> URL:

[jira] [Updated] (HADOOP-13329) Dockerfile doesn't work on Linux/ppc

2016-07-06 Thread Allen Wittenauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-13329:
--
Status: Patch Available  (was: Open)

> Dockerfile doesn't work on Linux/ppc
> 
>
> Key: HADOOP-13329
> URL: https://issues.apache.org/jira/browse/HADOOP-13329
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Amir Sanjar
> Attachments: HADOOP-13329.patch
>
>
> We need to rework how the Dockerfile is built to support both Linux/x86 and 
> Linux/PowerPC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile

2016-07-06 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365427#comment-15365427
 ] 

Allen Wittenauer commented on HADOOP-13342:
---

The basic problem is that the package needs to be reliable for an extremely 
long length of time since eventually this Dockerfile will end up in a release.  
Requiring a change every 6 months (was it even that?) is simply not good 
enough.  With the build effectively being broken, it was much more expedient to 
just remove it rather than spend the effort to come up with a long term 
solution.  It's also worth pointing out that the PowerPC version of the 
Dockerfile (HADOOP-13329) will also not be including ISAL.

> ISAL download is breaking the Dockerfile
> 
>
> Key: HADOOP-13342
> URL: https://issues.apache.org/jira/browse/HADOOP-13342
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-13342.00.patch
>
>
> http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb
>  is returning a 404.  We need to replace or remove this hack to prevent this 
> from happening in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365418#comment-15365418
 ] 

Allen Wittenauer commented on HADOOP-13335:
---

bq. It is going to be hard to remove either of these. I don't know when 
something that has been deprecated in a previous release has actually been 
removed.

There have been a ton of removals all over the place in trunk already.  S3, 
hftp, ... lots and lots of places.  Bear in mind that the last opportunity 
Hadoop had to remove content was over five years ago.

bq. If hadoop jar and yarn jar should behave the same

Once again: they do not.  I'm not sure how many more and different ways I can 
state this point.  The rest of that discussion is pretty much moot since the 
first opportunity to make them function the same will be 4.x, easily years off.

bq. This is arguable - but printing a new warning in the 2.7.0 release can be 
considered to be an incompatible change.

Deprecation warnings have always been added to the output throughout the 
history of Hadoop.  Expect a lot more to show up in trunk.


> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer

2016-07-06 Thread Gregory Chanan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HADOOP-13346:

Attachment: HADOOP-13346.patch

Here's a patch with a test.  The test fails without the code changes and 
demonstrates that a Writer that checks whether it has been closed will cause a 
problem with the current code.

There are a number of ways to fix this, I simply added a configuration step 
that checks if any of jackson's JsonFactory features are configured, and in 
turn configures them for the DelegationTokenAuthenticationHandler.  This lets 
you specify arbitrary jackson features, which is nice, but it does have some 
risk in terms of feature names changing.  Alternatively, you could define a 
static mapping from features to auth handler configs, or allow the user to pass 
in a JsonFactory to init (although the code currently casts everything to 
String, so that would have to change).

> DelegationTokenAuthenticationHandler writes via closed writer
> -
>
> Key: HADOOP-13346
> URL: https://issues.apache.org/jira/browse/HADOOP-13346
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Gregory Chanan
>Priority: Minor
> Attachments: HADOOP-13346.patch
>
>
> By default, jackson's ObjectMapper closes the writer after writing, so in the 
> following code
> {code}
> ObjectMapper jsonMapper = new ObjectMapper();
> jsonMapper.writeValue(writer, map);
> writer.write(ENTER);
> {code}
> (https://github.com/apache/hadoop/blob/8a9d293dd60f6d51e1574e412d40746ba8175fe1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282)
> writer.write actually writes to a closed stream.  This doesn't seem to cause 
> a problem with the version of jetty that hadoop uses (those just ignore 
> closes), but causes problems on later verisons of jetty -- I hit this on 
> jetty 8 while implementing SOLR-9200.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer

2016-07-06 Thread Gregory Chanan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HADOOP-13346:

Status: Patch Available  (was: Open)

> DelegationTokenAuthenticationHandler writes via closed writer
> -
>
> Key: HADOOP-13346
> URL: https://issues.apache.org/jira/browse/HADOOP-13346
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Gregory Chanan
>Priority: Minor
> Attachments: HADOOP-13346.patch
>
>
> By default, jackson's ObjectMapper closes the writer after writing, so in the 
> following code
> {code}
> ObjectMapper jsonMapper = new ObjectMapper();
> jsonMapper.writeValue(writer, map);
> writer.write(ENTER);
> {code}
> (https://github.com/apache/hadoop/blob/8a9d293dd60f6d51e1574e412d40746ba8175fe1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282)
> writer.write actually writes to a closed stream.  This doesn't seem to cause 
> a problem with the version of jetty that hadoop uses (those just ignore 
> closes), but causes problems on later verisons of jetty -- I hit this on 
> jetty 8 while implementing SOLR-9200.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer

2016-07-06 Thread Gregory Chanan (JIRA)

Gregory Chanan created HADOOP-13346:
---

 Summary: DelegationTokenAuthenticationHandler writes via closed 
writer
 Key: HADOOP-13346
 URL: https://issues.apache.org/jira/browse/HADOOP-13346
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Gregory Chanan
Priority: Minor


By default, jackson's ObjectMapper closes the writer after writing, so in the 
following code
{code}
ObjectMapper jsonMapper = new ObjectMapper();
jsonMapper.writeValue(writer, map);
writer.write(ENTER);
{code}
(https://github.com/apache/hadoop/blob/8a9d293dd60f6d51e1574e412d40746ba8175fe1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282)

writer.write actually writes to a closed stream.  This doesn't seem to cause a 
problem with the version of jetty that hadoop uses (those just ignore closes), 
but causes problems on later verisons of jetty -- I hit this on jetty 8 while 
implementing SOLR-9200.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread Federico Czerwinski (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365356#comment-15365356
 ] 

Federico Czerwinski commented on HADOOP-13075:
--

Thanks Chris, will try that.

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library

2016-07-06 Thread Aaron T. Myers (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365355#comment-15365355
 ] 

Aaron T. Myers commented on HADOOP-11540:
-

Hi [~drankye], overall the latest patch looks pretty good to me. I have a few 
small comments, and I'll be +1 once these are addressed:

# Should use the Logger system for this message, instead of going directly to 
stdout, likewise for doEncodeByConvertingToDirectBuffers:
{code}
+System.out.println("WARNING: doDecodeByConvertingToDirectBuffers " +
+"is used, not efficiently");
{code}
# I think you mean "get/set" here (in two places in the code):
{code}
+  // Set/set and only used by native codes.
{code}
# I think that the constructor for {{ByteBufferEncodingState}} should perhaps 
have its argument renamed to "{{encodeLength}}" instead of "{{decodeLength}}"
# Per Colin's earlier suggestion, I think we still have some error checking to 
do in {{jni_common.c#setCoder}}, just like you've done in {{getCoder}}.
# A bit of a meta comment: there's so much overlap/repetition between all of 
the encoder and decoder classes tha
t I wonder if there isn't some good way of refactoring this code to combine 
encoding/decoding functionality in a
 single class. Maybe that would make things more complex, but as it stands 
there's effectively two copies of a l
ot of the code in this patch. If you'd like to ignore this piece of feedback 
for now, that's totally fine, but s
omething to think about.

> Raw Reed-Solomon coder using Intel ISA-L library
> 
>
> Key: HADOOP-11540
> URL: https://issues.apache.org/jira/browse/HADOOP-11540
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Kai Zheng
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, 
> HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v2.patch, 
> HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, 
> HADOOP-11540-v7.patch, HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, 
> HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel 
> ISAL-v1.pdf
>
>
> This is to provide RS codec implementation using Intel ISA-L library for 
> encoding and decoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

2016-07-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365342#comment-15365342
 ] 

ASF GitHub Bot commented on HADOOP-12527:
-

Github user ejono commented on the issue:

https://github.com/apache/hadoop/pull/39
  
@benmccann, is that actually necessary for this patch? I don't actually 
know. Also, that line you point to currently has an open-ended max version, so 
it's already going to use the latest, right?


> Upgrade Avro dependency to 1.7.7
> 
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365336#comment-15365336
 ] 

Chris Nauroth commented on HADOOP-13075:


[~fed...@gmail.com], I'm unable to reproduce that test failure.  Sometimes this 
kind of problem happens if a dependent module has changed but not rebuilt.  I 
suggest that you run a fresh {{mvn clean install -DskipTests 
-Dmaven.javadoc.skip=true}} at the root of the Hadoop source tree to install 
fresh builds of all sub-components in your local Maven repository.  Then, retry 
the test.

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling

2016-07-06 Thread Ryan Blue (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365275#comment-15365275
 ] 

Ryan Blue edited comment on HADOOP-12878 at 7/6/16 10:55 PM:
-

FileInputFormat works slightly differently. First, the [split 
size|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L445]
 is calculated from the file's reported block size and the current min and max 
split sizes. Then, [the file is broken into N 
splits|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L410-416]
 that size, where {{N = Math.ceil(fileLength / splitSize)}}. The block 
locations are then used to determine [where each split is 
located|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L448],
 based on the split's starting offset.

The result is that {{getFileBlockLocations}} can return a single location for 
the entire file and you'll still end up with N roughly block-sized splits. This 
is what enables you to get more parallelism by setting smaller split sizes, 
even if the resulting splits don't correspond to different blocks. In our 
environment, we use a 64MB S3 block size and don't see a bottleneck from one 
input split per file.


was (Author: rdblue):
FileInputFormat works slightly differently. First, the [split 
size|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L445]
 is calculated from the file's reported block size and the current min and max 
split sizes. Then, the file is broken into N splits that size, where {{N = 
Math.ceil(fileLength / splitSize)}}. The block locations are then used to 
determine where each split is located, based on the split's starting offset.

The result is that {{getFileBlockLocations}} can return a single location for 
the entire file and you'll still end up with N roughly block-sized splits. This 
is what enables you to get more parallelism by setting smaller split sizes, 
even if the resulting splits don't correspond to different blocks. In our 
environment, we use a 64MB S3 block size and don't see a bottleneck from one 
input split per file.

> Impersonate hosts in s3a for better data locality handling
> --
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Currently, {{localhost}} is passed as locality for each block, causing all 
> blocks involved in job to initially target the same node (RM), before being 
> moved by the scheduler (to a rack-local node). This reduces parallelism for 
> jobs (with short-lived mappers). 
> We should mimic Azures implementation: a config setting 
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list 
> of hostnames in the cluster to return to {{getFileBlockLocations}}. 
> Possible optimization: for larger systems, it might be better to return N 
> (5?) random hostnames to prevent passing a huge array (the downstream code 
> assumes size = O(3)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling

2016-07-06 Thread Ryan Blue (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365275#comment-15365275
 ] 

Ryan Blue commented on HADOOP-12878:


FileInputFormat works slightly differently. First, the [split 
size|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#L445]
 is calculated from the file's reported block size and the current min and max 
split sizes. Then, the file is broken into N splits that size, where {{N = 
Math.ceil(fileLength / splitSize)}}. The block locations are then used to 
determine where each split is located, based on the split's starting offset.

The result is that {{getFileBlockLocations}} can return a single location for 
the entire file and you'll still end up with N roughly block-sized splits. This 
is what enables you to get more parallelism by setting smaller split sizes, 
even if the resulting splits don't correspond to different blocks. In our 
environment, we use a 64MB S3 block size and don't see a bottleneck from one 
input split per file.

> Impersonate hosts in s3a for better data locality handling
> --
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Currently, {{localhost}} is passed as locality for each block, causing all 
> blocks involved in job to initially target the same node (RM), before being 
> moved by the scheduler (to a rack-local node). This reduces parallelism for 
> jobs (with short-lived mappers). 
> We should mimic Azures implementation: a config setting 
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list 
> of hostnames in the cluster to return to {{getFileBlockLocations}}. 
> Possible optimization: for larger systems, it might be better to return N 
> (5?) random hostnames to prevent passing a huge array (the downstream code 
> assumes size = O(3)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread Federico Czerwinski (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365253#comment-15365253
 ] 

Federico Czerwinski commented on HADOOP-13075:
--

When I run {{mvn test}} in {{branch-2}}, without my patch, I get the following 
error (with is the same error I'm getting with my patch on):

{code}
testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.TestS3AMiniYarnCluster)  Time 
elapsed: 4.112 sec  <<< ERROR!
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerHealth.(SchedulerHealth.java:78)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.(CapacityScheduler.java:233)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createScheduler(ResourceManager.java:386)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:575)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1001)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:290)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:321)
at 
org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:115)
at 
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:461)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:289)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.fs.s3a.yarn.TestS3AMiniYarnCluster.beforeTest(TestS3AMiniYarnCluster.java:64)
{code}

any ideas? this is with version {{c3d9ac8}}, which is from about an hour.

cheers

Fede

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365228#comment-15365228
 ] 

Thomas Poepping commented on HADOOP-13344:
--

That was my mistake, I didn't mean to include 2.7.2 as a target version, only 
2.8.0.

I found while writing this that we could use 'jar tf ${jarname}', but that was 
always much slower than grepping the jarfile directly.

As far as documentation, I don't have any plans. I assume that any 
documentation that already exists regarding hadoop-config.sh should have this 
feature documented. As far as client applications have documentation, I don't 
see any good way to do it. We can't very well change the documentation of each 
application depending on Hadoop. It could also be added to a hadoop-config help 
method, if one exists. I'll look into it.

First and foremost though -- I'll be targeting trunk. The reason I targeted 
this directly is because the structure of the hadoop-config file is very 
different in trunk.

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Poepping updated HADOOP-13344:
-
Target Version/s: 2.8.0  (was: 2.8.0, 2.7.3)

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365211#comment-15365211
 ] 

Sean Busbey commented on HADOOP-13344:
--

thanks for working on this. I was just complaining about the inclusion of slf4j 
bindings in this classpath earlier today. :)

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365206#comment-15365206
 ] 

Thomas Poepping commented on HADOOP-13344:
--

Thanks for the quick response -- I'm working on a patch that targets trunk as 
well

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HADOOP-13344:
-
Issue Type: New Feature  (was: Improvement)

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HADOOP-13344:
-
Component/s: scripts

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365208#comment-15365208
 ] 

Sean Busbey commented on HADOOP-13344:
--

-1 (non-binding) on this going into 2.7.z. Maintenance releases aren't the 
place to change the dependency set, even if it's opt-in. It invites folks to 
rely on behavior that they won't be able to use if they have to revert to an 
earlier maintenance release.

How are we planning to document this as an option for the client classpath? 
Will operators have to have looked at the source of our scripts to know that 
this feature exists?

{code}
+  find $parent_directory -maxdepth 1 -name "*.jar" -exec grep -HLs \
+  "org/slf4j/impl/StaticLoggerBinder.class" {} \; \
+  | paste -sd ':' -
{code}

At this point in bootstrapping, do we have access to a JRE that we could use to 
enumerate jar contents (or some zip accessor)? that'd be more reliable than 
greping the jar file directly.


> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin, scripts
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt

2016-07-06 Thread Andrew Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365201#comment-15365201
 ] 

Andrew Wang commented on HADOOP-12893:
--

So one thing I could use clarification is if we need to include NOTICE entries 
if the downstream project has incorrectly constructed their NOTICE file, or 
we've already captured their transitive dependencies.

* Jetty spends a lot of space listing out their deps and licenses, but that's 
supposed to go in LICENSE, not NOTICE. We also already look at all of Jetty's 
transitive dependencies ourselves.
* Xerces is another Apache project, so we don't need the Apache copyright. Also 
considering the Xerces code was donated to Apache, are they legally required to 
list the IBM/Sun/iClick copyright attributions?
* None of snappy-java's entries look like legally required attributions either

The jcip changes look good though, nice catch.

> Verify LICENSE.txt and NOTICE.txt
> -
>
> Key: HADOOP-12893
> URL: https://issues.apache.org/jira/browse/HADOOP-12893
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Xiao Chen
>Priority: Blocker
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, 
> HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, 
> HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, 
> HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, 
> HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, 
> HADOOP-12893.addendum-trunk.01.patch, HADOOP-12893.branch-2.01.patch, 
> HADOOP-12893.branch-2.6.01.patch, HADOOP-12893.branch-2.7.01.patch, 
> HADOOP-12893.branch-2.7.02.patch, HADOOP-12893.branch-2.7.3.01.patch
>
>
> We have many bundled dependencies in both the source and the binary artifacts 
> that are not in LICENSE.txt and NOTICE.txt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365193#comment-15365193
 ] 

Sean Busbey commented on HADOOP-13344:
--

But you should make a patch that targets master first, since changes need to 
land there before going back to 2.y

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365192#comment-15365192
 ] 

Sean Busbey commented on HADOOP-13344:
--

see this section on the wiki page linked above:

https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13340) Compress Hadoop Archive output

2016-07-06 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365180#comment-15365180
 ] 

Jason Lowe commented on HADOOP-13340:
-

A Hadoop archive (har) helps solve the small file problem by combining many 
files into a few files, along with an index file to help find the original 
files.  Therefore the data for most original files will appear at some non-zero 
offset within one of the har files.  A gzip stream cannot be decoded at an 
arbitrary offset within the stream, since the symbols in the stream are 
relative to what's appears in the stream before it.  So if we used the gzip 
codec to compress those har files then that disables the ability to seek 
arbitrarily within the har file.  The client would need to uncompress the 
entire file up to the point where the original file data starts which would 
make them very slow to access.

The seek problem can be solved by compressing the original files as separate 
compression streams within the larger har file, but arguably that can be done 
today by compressing the original files before adding them to the har archive.


> Compress Hadoop Archive output
> --
>
> Key: HADOOP-13340
> URL: https://issues.apache.org/jira/browse/HADOOP-13340
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: 2.5.0
>Reporter: Duc Le Tu
>  Labels: features, performance
>
> Why Hadoop Archive tool cannot compress output like other map-reduce job? 
> I used some options like -D mapreduce.output.fileoutputformat.compress=true 
> -D 
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec
>  but it's not work. Did I wrong somewhere?
> If not, please support option for compress output of Hadoop Archive tool, 
> it's very neccessary for data retention for everyone (small files problem and 
> compress data).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

2016-07-06 Thread Ben McCann (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365160#comment-15365160
 ] 

Ben McCann edited comment on HADOOP-12527 at 7/6/16 9:38 PM:
-

+1 (although I'd really like to see Avro upgraded to 1.8.x)


was (Author: chengas123):
+1 (although I'd really like to see Avro upgraded to 1.8.x)

There's a patch available for this: https://github.com/apache/hadoop/pull/39

> Upgrade Avro dependency to 1.7.7
> 
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

2016-07-06 Thread Ben McCann (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365160#comment-15365160
 ] 

Ben McCann edited comment on HADOOP-12527 at 7/6/16 9:38 PM:
-

+1 (although I'd really like to see Avro upgraded to 1.8.x)

There's a patch available for this: https://github.com/apache/hadoop/pull/39


was (Author: chengas123):
+1 (although I'd really like to see Avro upgraded to 1.8.x)

> Upgrade Avro dependency to 1.7.7
> 
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

2016-07-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365164#comment-15365164
 ] 

ASF GitHub Bot commented on HADOOP-12527:
-

Github user benmccann commented on the issue:

https://github.com/apache/hadoop/pull/39
  
Also upgrade avro-maven-plugin? 
https://github.com/apache/hadoop/blob/trunk/pom.xml#L225


> Upgrade Avro dependency to 1.7.7
> 
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12527) Upgrade Avro dependency to 1.7.7

2016-07-06 Thread Ben McCann (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365160#comment-15365160
 ] 

Ben McCann commented on HADOOP-12527:
-

+1 (although I'd really like to see Avro upgraded to 1.8.x)

> Upgrade Avro dependency to 1.7.7
> 
>
> Key: HADOOP-12527
> URL: https://issues.apache.org/jira/browse/HADOOP-12527
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 2.7.1
>Reporter: Jonathan Kelly
>
> Hadoop has depended upon Avro 1.7.4 for a couple of years now (see 
> HADOOP-9672), but Apache Spark depends upon what is currently the latest 
> version of Avro (1.7.7).
> This can cause issues if Spark is configured to include the full Hadoop 
> classpath, as the classpath would then contain both Avro 1.7.4 and 1.7.7, 
> with the 1.7.4 classes possibly winning depending on ordering. Here is an 
> example of this issue: 
> http://stackoverflow.com/questions/33159254/avro-error-on-aws-emr/33403111#33403111
> Would it be possible to upgrade Hadoop's Avro dependency to 1.7.7 now?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers

2016-07-06 Thread Ray Chiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365146#comment-15365146
 ] 

Ray Chiang commented on HADOOP-13254:
-

I'm good either way, as long as the Javadoc API is clearly documented for how 
it should be used.  I'm not sure it's always good to have a threaded version 
(otherwise, we'll have a thread, creating a thread, creating a thread, ad 
nauseum).  So, I see two options:

- Create a single API that supports threaded disk checking and a separate API 
that supports non-threaded, direct call disk checking.
- Create a single API that supports both.


> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, 
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, 
> HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2016-07-06 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365126#comment-15365126
 ] 

Chris Nauroth commented on HADOOP-13345:


[~andrew.wang], [~jzhuge] and [~eddyxu], please have a look.  Feedback 
appreciated.  I will be offline for a while, returning 7/17, so I won't be able 
to respond to comments until then.

> S3Guard: Improved Consistency for S3A
> -
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch, 
> S3GuardImprovedConsistencyforS3A.pdf
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a 
> stronger consistency model than what is currently offered.  The solution 
> coordinates with a strongly consistent external store to resolve 
> inconsistencies caused by the S3 eventual consistency model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13342) ISAL download is breaking the Dockerfile

2016-07-06 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365121#comment-15365121
 ] 

Ravi Prakash edited comment on HADOOP-13342 at 7/6/16 9:25 PM:
---

Oh! Sorry about that Kai. Please create a new JIRA and I'm happy to +1 it. 
Also, I thought the Dockerfile was Allen's work. Didn't realize you'd added to 
it. My apologies.


was (Author: raviprak):
Oh! Sorry about that Kai. Please create a new JIRA and I'm happy to +1 it.

> ISAL download is breaking the Dockerfile
> 
>
> Key: HADOOP-13342
> URL: https://issues.apache.org/jira/browse/HADOOP-13342
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-13342.00.patch
>
>
> http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb
>  is returning a 404.  We need to replace or remove this hack to prevent this 
> from happening in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2016-07-06 Thread Chris Nauroth (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-13345:
---
Attachment: HADOOP-13345.prototype1.patch
S3GuardImprovedConsistencyforS3A.pdf

I'm attaching a design document and a prototype patch that I've been testing.  
The design document describes the proposed solution in more detail.  To 
summarize the prototype patch:

* There is a new module: hadoop-tools/hadoop-s3guard.  It was convenient to 
structure the prototype this way, isolated in a separate module, to avoid 
conflicts with the ongoing hadoop-aws work.  I'm not certain this is the best 
permanent structure.  It might make more sense to embed it all within 
hadoop-aws.  There are pros and cons either way.
* The hadoop-s3guard module defines a {{ConsistentS3AFileSystem}} class, which 
is a subclass of {{S3AFileSystem}}.  This is the main entry point for this 
work.  It overrides several methods to coordinate with the consistent external 
store.
* {{ConsistentStore}} defines the interface expected from the strongly 
consistent store.  There is currently one implementation using DynamoDB as the 
back-end: {{DynamoDBConsistentStore}}.
* Even though it's a prototype, I wrote JavaDocs to explain what is happening 
at the code level.  I think the JavaDocs serve as a good companion piece to the 
design doc.
* For testing, I have used some Maven trickery to reuse all S3A test suites 
from hadoop-aws within hadoop-s3guard.  src/test/resources/core-site.xml 
rewires the s3a: scheme so that all tests are executing against 
{{ConsistentS3AFileSystem}}.  All of these tests are passing currently.  This 
provides basic coverage of the hadoop-s3guard code paths and confirmation that 
the work hasn't yet introduced regressions.
* Currently missing is any form of dedicated testing that specifically tries to 
trigger eventually consistent behavior and confirm that S3Guard handles it 
successfully.  This means that we don't yet have automated testing that really 
confirms S3Guard does what it needs to do.  Since eventually consistent 
behavior is non-deterministic, I think we'll need to explore mock-based 
approaches that simulate eventual consistency by returning incomplete/out-dated 
results on the first try, and then complete results on subsequent retries.  
We'll need to combine this with tests that really integrate with S3.
* Also currently missing is any form of end user documentation.
* I have marked various TODOs in the code to indicate important work items that 
remain to be done.


> S3Guard: Improved Consistency for S3A
> -
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch, 
> S3GuardImprovedConsistencyforS3A.pdf
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a 
> stronger consistency model than what is currently offered.  The solution 
> coordinates with a strongly consistent external store to resolve 
> inconsistencies caused by the S3 eventual consistency model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile

2016-07-06 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365121#comment-15365121
 ] 

Ravi Prakash commented on HADOOP-13342:
---

Oh! Sorry about that Kai. Please create a new JIRA and I'm happy to +1 it.

> ISAL download is breaking the Dockerfile
> 
>
> Key: HADOOP-13342
> URL: https://issues.apache.org/jira/browse/HADOOP-13342
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-13342.00.patch
>
>
> http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb
>  is returning a 404.  We need to replace or remove this hack to prevent this 
> from happening in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2016-07-06 Thread Chris Nauroth (JIRA)

Chris Nauroth created HADOOP-13345:
--

 Summary: S3Guard: Improved Consistency for S3A
 Key: HADOOP-13345
 URL: https://issues.apache.org/jira/browse/HADOOP-13345
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs/s3
Reporter: Chris Nauroth
Assignee: Chris Nauroth


This issue proposes S3Guard, a new feature of S3A, to provide an option for a 
stronger consistency model than what is currently offered.  The solution 
coordinates with a strongly consistent external store to resolve 
inconsistencies caused by the S3 eventual consistency model.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365040#comment-15365040
 ] 

Hadoop QA commented on HADOOP-13344:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  4s{color} 
| {color:red} HADOOP-13344 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12816498/HADOOP-13344.patch |
| JIRA Issue | HADOOP-13344 |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9931/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Poepping updated HADOOP-13344:
-
Attachment: HADOOP-13344.patch

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.8.0, 2.7.2
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-13344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Poepping updated HADOOP-13344:
-
Affects Version/s: 2.8.0
   Status: Patch Available  (was: Open)

> Add option to exclude Hadoop's SLF4J binding
> 
>
> Key: HADOOP-13344
> URL: https://issues.apache.org/jira/browse/HADOOP-13344
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: bin
>Affects Versions: 2.7.2, 2.8.0
>Reporter: Thomas Poepping
>  Labels: patch
> Attachments: HADOOP-13344.patch
>
>
> If another application that uses the Hadoop classpath brings in its own SLF4J 
> binding for logging, and that jar is not the exact same as the one brought in 
> by Hadoop, then there will be a conflict between logging jars between the two 
> classpaths. This patch introduces an optional setting to remove Hadoop's 
> SLF4J binding from the classpath, to get rid of this problem.
> This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
> has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13344) Add option to exclude Hadoop's SLF4J binding

2016-07-06 Thread Thomas Poepping (JIRA)

Thomas Poepping created HADOOP-13344:


 Summary: Add option to exclude Hadoop's SLF4J binding
 Key: HADOOP-13344
 URL: https://issues.apache.org/jira/browse/HADOOP-13344
 Project: Hadoop Common
  Issue Type: Improvement
  Components: bin
Affects Versions: 2.7.2
Reporter: Thomas Poepping


If another application that uses the Hadoop classpath brings in its own SLF4J 
binding for logging, and that jar is not the exact same as the one brought in 
by Hadoop, then there will be a conflict between logging jars between the two 
classpaths. This patch introduces an optional setting to remove Hadoop's SLF4J 
binding from the classpath, to get rid of this problem.

This patch should be applied to 2.8.0, as bin/ and hadoop-config.sh structure 
has been changed in 3.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats

2016-07-06 Thread Jitendra Nath Pandey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365021#comment-15365021
 ] 

Jitendra Nath Pandey commented on HADOOP-13283:
---

+1 for the latest patch.

> Support reset operation for new global storage statistics and per FS storage 
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch, 
> HADOOP-13283.002.patch, HADOOP-13283.003.patch, HADOOP-13283.004.patch
>
>
> Applications may reuse the file system object across jobs and its storage 
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports 
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Siddharth Seth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364973#comment-15364973
 ] 

Siddharth Seth commented on HADOOP-13335:
-

bq. It's going to be hard to take yarn jar or hadoop jar away. It's doubtful 
they will ever get removed. That said, we can at least make them act and work 
the same way. To me, that's the ultimate goal and it's pretty close to what 
happens in trunk:
1. yarn command sucks in yarn-env.sh, hadoop-env.sh, yarn-config.sh and 
hadoop-config.sh in a way that should be mostly conflict-free. (non-yarn 
commands do not pull in yarn-x.sh, obviously)
2. If YARN_OPTS is defined, yarn x (jar, rmadmin, etc) will use it but throw a 
deprecation warning.
3. Otherwise use HADOOP_OPTS

It is going to be hard to remove either of these. I don't know when something 
that has been deprecated in a previous release has actually been removed.
If hadoop jar and yarn jar should behave the same - and 1) 'yarn jar' is not 
the preferred usage, or 2) 'yarn jar' behaves as an alias - I'd be in favor of 
removing this warnings altogether. Don't encourage users to use yarn jar over 
hadoop jar/ don't advertise yarn jar.

In terms of removing the warning - I'm all for it, and is the preferred 
approach to 'fix' this jar (that's the first suggestion in the description).

This is arguable - but printing a new warning in the 2.7.0 release can be 
considered to be an incompatible change. Other than being an annoyance to users 
- Hive silent mode is broken by this, since it was not written to work with 
output from the hadoop command.

Lets keep the change / detailed recommendation in the 'help' output - and 
remove it from hadoop jar invocation.

bq. Very long term (post-3.x), it would probably be better if hive called 
hadoop-config.sh and/or hadoop-functions.sh directly. This would bypass the 
middleman and give much better control. I'd be very interested to hear what 
sort of holes we have in the functionality here that makes this 
hard/impossible. Off the top, I suspect we need to make one big function of the 
series of function calls in hadoop-config.sh, but would love to hear your 
insight on this.
I don't know enough about the internals of the scripts to have an educated 
opinion on this. hadoop scripts, required environment variables, how they 
interact seems quite complicated. Setting HADOOP_CLIENT_OPTS is apparently the 
way to change the hiveclient heap size. I would expect hive to control this 
independently, and not depend on variables exported by hadoop scripts. One 
possible usage is for Hadoop to provide basic information - CLASSPATH, configs. 
Products like Hive build on top of this information, rather than trying to use 
hadoop scripts, and define their own mechanism for users to specify various 
environment variables. hadoop jar is obviously useful for custom tools, which 
want a simple way to execute on a hadoop cluster.

> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364885#comment-15364885
 ] 

Allen Wittenauer commented on HADOOP-13335:
---

bq. we are well past the point where we can remove "hadoop jar" or "yarn jar" 
anytime in the future.

Totally agree and that was never really on the table.  Long term, hadoop jar == 
yarn jar functionality-wise, just not in the 2.x/3.x timeframe.

bq. How about we simply remove the warnings in "hadoop jar" (kind of how we 
'undeprecated' the mapred API) and move forward?

Because users don't understand that YARN_OPTS and HADOOP_OPTS are separate in 
branch-2.  This whole situation started a while back because users were 
changing/setting YARN_OPTS and didn't understand why hadoop jar blah didn't 
pick up their changes when they were running YARN jobs.

> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364862#comment-15364862
 ] 

Vinod Kumar Vavilapalli commented on HADOOP-13335:
--

Folks, we are well past the point where we can remove "hadoop jar" or "yarn 
jar" anytime in the future. Our users know about them, and I'd not even think 
about deprecating / removing them in 3.x / 4.x or any up-version for that 
matter.

How about we simply remove the warnings in "hadoop jar" (kind of how we 
'undeprecated' the mapred API) and move forward?

> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers

2016-07-06 Thread Robert Kanter (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364772#comment-15364772
 ] 

Robert Kanter commented on HADOOP-13254:


Not just future contributors, but users might make their own disk validator 
plugins that they don't contribute back, which can do really anything.  While 
we don't need to protect against everything (e.g. I think it's reasonable to 
assume they shouldn't call {{System.exit()}}), we should expect some sort of 
initialization that might be done in the Constructor.  

{quote}The cheapest solution I see is to use locks.{quote}
Another option is to add {{init()}} and {{close()}} methods, similar to what we 
have in the {{Service}} classes.  This way, I think it's okay to assume that 
users should do things like starting threads or similar things in the 
{{init()}} method, instead of the constructor; and we give them a nice place to 
clean things up in the {{close()}} method.

> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, 
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, 
> HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13233) help of stat is confusing

2016-07-06 Thread Akira Ajisaka (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364763#comment-15364763
 ] 

Akira Ajisaka commented on HADOOP-13233:


Hi [~shenyinjie], you can take it over. Thanks.

> help of stat is confusing
> -
>
> Key: HADOOP-13233
> URL: https://issues.apache.org/jira/browse/HADOOP-13233
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Xiaohe Lan
>Priority: Trivial
>  Labels: newbie
>
> %b is actually printing the size of a file in bytes, while in help it says 
> filesize in blocks.
> {code}
> hdfs dfs -help stat
> -stat [format]  ... :
>   Print statistics about the file/directory at 
>   in the specified format. Format accepts filesize in
>   blocks (%b)
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Akira Ajisaka (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364756#comment-15364756
 ] 

Akira Ajisaka commented on HADOOP-11993:


I ran {{mvn clean verify -DskipTests}} and it failed.
{noformat}
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (depcheck) @ hadoop-common ---
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BannedDependencies failed 
with message:
Found Banned Dependency: com.sun.jersey:jersey-servlet:jar:1.19
Found Banned Dependency: com.sun.jersey:jersey-json:jar:1.19
Use 'mvn dependency:tree' to locate the source of the banned dependencies.
{noformat}
Would you fix the failure? In addition, would you fix the indent?

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13338) Incompatible change to SortedMapWritable

2016-07-06 Thread Akira Ajisaka (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364662#comment-15364662
 ] 

Akira Ajisaka commented on HADOOP-13338:


I'm +1 for reverting HADOOP-10465 from branch-2 and branch-2.8.

> Incompatible change to SortedMapWritable
> 
>
> Key: HADOOP-13338
> URL: https://issues.apache.org/jira/browse/HADOOP-13338
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Siddharth Seth
>Priority: Critical
>
> Hive does not compile against Hadoop-2.8.0-SNAPSHOT
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) 
> on project hive-contrib: Compilation failure
> [ERROR] 
> /Users/sseth/work2/projects/hive/dev/forMvnInstall/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableOutput.java:[215,70]
>  incompatible types: java.lang.Object cannot be converted to 
> java.util.Map.Entry
> {code}
> Looks like the change in HADOOP-10465 causes this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13343) globStatus returns null for file path that exists but is filtered

2016-07-06 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364597#comment-15364597
 ] 

Jason Lowe commented on HADOOP-13343:
-

For example, the following code in MapReduce input split calculation expects 
null to mean the file doesn't exist but an empty array to mean it exists but 
doesn't match any filters:
{code}
  private List singleThreadedListStatus(JobContext job, Path[] dirs,
  PathFilter inputFilter, boolean recursive) throws IOException {
List result = new ArrayList();
List errors = new ArrayList();
for (int i=0; i < dirs.length; ++i) {
  Path p = dirs[i];
  FileSystem fs = p.getFileSystem(job.getConfiguration()); 
  FileStatus[] matches = fs.globStatus(p, inputFilter);
  if (matches == null) {
errors.add(new IOException("Input path does not exist: " + p));
  } else if (matches.length == 0) {
errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
  } else {
{code}

There was a case where a user passed the path to a _SUCCESS file as one of the 
input paths to a job, and the hidden file filter in FileInputFormat suppressed 
the _SUCCESS file.  globStatus returning null instead of an empty array 
triggered the code to report a misleading error to the user.


> globStatus returns null for file path that exists but is filtered
> -
>
> Key: HADOOP-13343
> URL: https://issues.apache.org/jira/browse/HADOOP-13343
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Priority: Minor
>
> If a file path without globs is passed to globStatus and the file exists but 
> the specified input filter suppresses it then globStatus will return null 
> instead of an empty array.  This makes it impossible for the caller to 
> discern the difference between the file not existing at all vs. being 
> suppressed by the filter and is inconsistent with the way it handles globs 
> for an existing dir but fail to match anything within the dir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13343) globStatus returns null for file path that exists but is filtered

2016-07-06 Thread Jason Lowe (JIRA)

Jason Lowe created HADOOP-13343:
---

 Summary: globStatus returns null for file path that exists but is 
filtered
 Key: HADOOP-13343
 URL: https://issues.apache.org/jira/browse/HADOOP-13343
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.7.2
Reporter: Jason Lowe
Priority: Minor


If a file path without globs is passed to globStatus and the file exists but 
the specified input filter suppresses it then globStatus will return null 
instead of an empty array.  This makes it impossible for the caller to discern 
the difference between the file not existing at all vs. being suppressed by the 
filter and is inconsistent with the way it handles globs for an existing dir 
but fail to match anything within the dir.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile

2016-07-06 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364560#comment-15364560
 ] 

Kai Zheng commented on HADOOP-13342:


Hi guys,

Why the support was simply removed without any sync up or notifying? If you 
don't like ping, probably you could do a simple check and will obviously find 
in http://http.us.debian.org/debian/pool/main/libi/libisal/ that the library 
was upgraded. The fix would be rather simple by changing the revision number 
({{2.16}}), instead of removing block of codes.

As we discussed somewhere, please notify the relevant guys when you revert 
someone's work. Thanks!

> ISAL download is breaking the Dockerfile
> 
>
> Key: HADOOP-13342
> URL: https://issues.apache.org/jira/browse/HADOOP-13342
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-13342.00.patch
>
>
> http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb
>  is returning a 404.  We need to replace or remove this hack to prevent this 
> from happening in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Allen Wittenauer (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364457#comment-15364457
 ] 

Allen Wittenauer commented on HADOOP-13335:
---

bq. Not yet - until someone feels the need to have hdfs used as a separate 
command to do something other than filesystem operations. (Really hope there's 
no hdfs jar command)

Not on my watch. :) It's pretty clear from this JIRA just how much damage the 
extra YARN stuff has caused.  (and this is just the tip of the iceberg...)  
Plus, between HADOOP-11485, HADOOP-12930, and a handful of other features, 
there are much better ways for 1st, 2nd, and 3rd parties to integrate extra 
stuff. \[1]

bq.  I don't see the difference between hadoop jar and yarn jar other than a 
different set of variables being set and respected by the different commands. 

That's correct. But those extra vars make a world of difference .  In branch-2, 
HADOOP\_OPTS and YARN\_OPTS don't cross.  Ever.  This effectively makes the 
hadoop, hdfs, and mapred entry points configured differently than yarn.

bq. Stepping back - should the YARN\_\* parameters exist?, and should yarn jar 
exist? If I understand you correctly, I think you're trying to get rid of some 
of this.

That's absolutely correct: none of this should exist and I've worked really 
hard on either removing or hiding the complexity going forward. But this only 
gets easier in trunk.  It's way too late and way too hard to fix this mess in 
branch-2.

bq. If 'yarn jar' is something that we think is confusing, or something we 
potentially want to get rid off - I'd say it's better to not print any warning 
at all - and leave hadoop jar as is?

It's going to be hard to take yarn jar or hadoop jar away. It's doubtful they 
will ever get removed. That said, we can at least make them act and work the 
same way.  To me, that's the ultimate goal and it's pretty close to what 
happens in trunk:

1. yarn command sucks in yarn-env.sh, hadoop-env.sh, yarn-config.sh and 
hadoop-config.sh in a way that should be mostly conflict-free. (non-yarn 
commands do not pull in yarn-x.sh, obviously)
2. If YARN\_OPTS is defined, yarn x (jar, rmadmin, etc) will use it but throw a 
deprecation warning.
3. Otherwise use HADOOP\_OPTS

As folks migrate to a release based on trunk, this extra fluff will go away at 
least configuration-wise.

Eventually, when we can remove support for all of these deprecated vars, this 
will reduce code complexity and gives us (effectively) one code path to test. 
But bear in mind that we're years away before YARN\_OPTS and friends disappear. 
 It's been 5+ years since our last trunk release. I'll likely be dead by the 
time 4.x comes out and these useless YARN\_\* vars can get culled officially.  
But the work has to start now.

bq. The hive binary could unset YARN_OPTS / YARN_CLIENT_OPTS - and leave them 
intact for the session/shell from where the hive binary was invoked.

Which, again, if hive wants to stick to using hadoop jar, this would be my 
advice. \[2]  Just keep in mind this also means that any user settings that 
they might have wanted to apply to their YARN environment will not kick in. 
It's no different than what is happening today, but it may not reflect what 
users want.  Thus we're back to why the warnings went in.

\[1]  I'm not saying the community would do this, but let's use hive as an 
example here of how much more powerful things are in trunk.  With HADOOP-12930, 
it's now possible for hive to add a 'hadoop hive' command or a (even more 
outrageous!) 'hdfs hivefs' command.  Rather than integrate *outside* the 
framework, one could integrate *inside* and pull exactly the information 
required. This will hopefully put 'hadoop jar' and 'yarn jar' on life support. 
We'll keep them around but it's going to be much more desirable to just 
integrate directly.  The new 'mapred streaming' command is a great example 
here: why make users call hadoop jar with some weird version number when it's 
now trivial to dynamically add commands at build or configure time?

\[2] Very long term (post-3.x), it would probably be better if hive called 
hadoop-config.sh and/or hadoop-functions.sh directly.  This would bypass the 
middleman and give much better control.  I'd be very interested to hear what 
sort of holes we have in the functionality here that makes this 
hard/impossible. Off the top, I suspect we need to make one big function of the 
series of function calls in hadoop-config.sh, but would love to hear your 
insight on this.

> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>

[jira] [Commented] (HADOOP-13254) Create framework for configurable disk checkers

2016-07-06 Thread Daniel Templeton (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364333#comment-15364333
 ] 

Daniel Templeton commented on HADOOP-13254:
---

[~rkanter], if an instance is creating threads in the constructor, that would 
make it a very naughty instance indeed.  Without an API that allows for a clean 
shutdown, creating threads is a risky proposition.  I agree with [~yufeigu] 
that when it comes to adding threads, whoever wants to add threads should 
change the API to include start and stop methods.  Since we can't make any 
guarantees about what future contributors will do, I also agree that being more 
defensive isn't a bad idea.  The cheapest solution I see is to use locks.

> Create framework for configurable disk checkers
> ---
>
> Key: HADOOP-13254
> URL: https://issues.apache.org/jira/browse/HADOOP-13254
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: util
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: HADOOP-13254.001.patch, HADOOP-13254.002.patch, 
> HADOOP-13254.003.patch, HADOOP-13254.004.patch, HADOOP-13254.005.patch, 
> HADOOP-13254.006.patch, HADOOP-13254.007.patch, HADOOP-13254.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread Andrew Olson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364327#comment-15364327
 ] 

Andrew Olson commented on HADOOP-13075:
---

The pull request looks good from here. Thanks!

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363896#comment-15363896
 ] 

Hadoop QA commented on HADOOP-11993:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
6s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  8m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12816374/HADOOP-11993.001.patch
 |
| JIRA Issue | HADOOP-11993 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  |
| uname | Linux d5bc49dca1fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / d792a90 |
| Default Java | 1.8.0_91 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9930/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9930/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11993:

Status: Patch Available  (was: Open)

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11993:

Target Version/s: 3.0.0-alpha1

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363880#comment-15363880
 ] 

Tsuyoshi Ozawa commented on HADOOP-11993:
-

{quote}
+   org.ow2.asm:asm:5.0.0
{quote}

BTW, I specified minimum version which JDK8 is supported. Hence, I specified 
5.0.0 for asm.

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11993:

Attachment: HADOOP-11993.001.patch

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11993:

Attachment: (was: HADOOP-11993.001.patch)

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11993:

Attachment: HADOOP-11993.001.patch

Attaching a first patch.

I tested with following command:
{quote}
mvn clean verify -Djersey.version=1.18 -DskipTests
...
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (depcheck) @ hadoop-common ---
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BannedDependencies failed 
with message:
Found Banned Dependency: com.sun.jersey:jersey-server:jar:1.18
Found Banned Dependency: com.sun.jersey:jersey-json:jar:1.18
Found Banned Dependency: com.sun.jersey:jersey-servlet:jar:1.18
Found Banned Dependency: com.sun.jersey:jersey-core:jar:1.18
{quote}

> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to 
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+ 
> codebase isn't pulling in any incompatible binaries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread Federico Czerwinski (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363862#comment-15363862
 ] 

Federico Czerwinski commented on HADOOP-13075:
--

hey [~cnauroth], as you can see I've created the PR for trunk, I'm working on 
the PR for branch-2 now. Cheers

> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2016-07-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363845#comment-15363845
 ] 

ASF GitHub Bot commented on HADOOP-13075:
-

GitHub user fedecz opened a pull request:

https://github.com/apache/hadoop/pull/110

[HADOOP-13075] Adding support for SSE-KMS, SSE-C and SSE-S3

https://issues.apache.org/jira/browse/HADOOP-13075

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fedecz/hadoop HADOOP-13075

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #110


commit bd465b90db9ea01c2076cab1d21cb9deb3424536
Author: Federico Czerwinski 
Date:   2016-06-29T00:03:52Z

[HADOOP-13075] Adding support for SSE-KMS and SSE-C




> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. 
> With native support in aws-java-sdk already available it should be fairly 
> straightforward [6],[7] to support the other two types of SSE with some 
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6] 
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12064) [JDK8] Update guice version to 4.0

2016-07-06 Thread Tsuyoshi Ozawa (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363840#comment-15363840
 ] 

Tsuyoshi Ozawa commented on HADOOP-12064:
-

[~pg1...@imperial.ac.uk] Thanks for the reporting. It looks strange to me since 
it exists in Guice 4.0. Anyway, I will check it on my local.

https://github.com/google/guice/blob/4.0/core/src/com/google/inject/Scopes.java#L184

> [JDK8] Update guice version to 4.0
> --
>
> Key: HADOOP-12064
> URL: https://issues.apache.org/jira/browse/HADOOP-12064
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
>  Labels: UpgradeKeyLibrary
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12064.001.patch, HADOOP-12064.002.WIP.patch, 
> HADOOP-12064.002.patch
>
>
> guice 3.0 doesn't work with lambda statement. 
> https://github.com/google/guice/issues/757
> We should upgrade it to 4.0 which includes the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it

2016-07-06 Thread Siddharth Seth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363835#comment-15363835
 ] 

Siddharth Seth commented on HADOOP-13335:
-

bq. There are no HDFS_* vars that change how the client operates. ...
Not yet - until someone feels the need to have hdfs used as a separate command 
to do something other than filesystem operations. (Really hope there's no hdfs 
jar command)

bq. I'm not sure there is more clarity required. If a user wants/needs YARN_*, 
then they need to use 'yarn jar'. 
Again - I don't see the difference between hadoop jar and yarn jar other than a 
different set of variables being set and respected by the different commands. 
Stepping back - should the YARN_* parameters exist?, and should yarn jar exist? 
If I understand you correctly, I think you're trying to get rid of some of this.
If 'yarn jar' is something that we think is confusing, or something we 
potentially want to get rid off - I'd say it's better to not print any warning 
at all - and leave hadoop jar as is?

The hive binary could unset YARN_OPTS / YARN_CLIENT_OPTS - and leave them 
intact for the session/shell from where the hive binary was invoked.
That said, the hive jira linked to this one proposes moving to using 'yarn jar' 
- and I believe this is mainly because of the WARNING. It would be good for 
users / downstream projects to know how this should be handled. Will yarn jar 
survive or not.
cc [~vinodkv] since this discussion has quite a bit to do with yarn, and to 
potentially get more context on yarn jar and the YARN_* variables.

> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, 
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, 
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' 
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive 
> uses it to start all it's services (HiveServer2, the hive client, beeline 
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of 
> sense - there's no relation to yarn other than requiring the classpath to 
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN 
> variables are set (leave it in the help message), or adding a mechanism which 
> would allow users to suppress this WARNING.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

76 matches

Mail list logo