[jira] [Created] (HADOOP-12956) Inevitable Log4j2 migration via slf4j

2016-03-22 Thread Gopal V (JIRA) 
Gopal V created HADOOP-12956:


 Summary: Inevitable Log4j2 migration via slf4j
 Key: HADOOP-12956
 URL: https://issues.apache.org/jira/browse/HADOOP-12956
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Gopal V


{{5 August 2015 --The Apache Logging Services™ Project Management Committee 
(PMC) has announced that the Log4j™ 1.x logging framework has reached its end 
of life (EOL) and is no longer officially supported.}}

https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces

A whole framework log4j2 upgrade has to be synchronized, partly for improved 
performance brought about by log4j2.

https://logging.apache.org/log4j/2.x/manual/async.html#Performance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Branch policy question

2016-03-22 Thread larry mccay 
Just to clarify, we are talking about a feature branch in which Allen and
others that are working on the branch could commit without requiring 3 +1’s.
Then, at some point, we will need 3 +1’s to merge the branch to trunk.
Correct?

I think that if we have a set of usecases that are being added and those
that are being reworked with test coverage for them all - maybe including
manual testing scenarios - that we could find folks that would be willing
to work with Allen to get it reviewed. I am listed as a branch committer.
If my vote would count then I would be more than willing to lend a hand
where I can - especially if we have the tests/instructions for testing.

I think that we need to strike a balance of not inhibiting progress and at
the same time not getting a patch so large that it can’t be reviewed.
Colin's point is valid and a design document that calls out the areas that
will need testing and general approach would help review volunteers.

There is always a risk with branches that the change gets to a level of
complexity that it is too hard to reasonably review. I think that the
community should allow folks to take that risk and try and provide enough
guidance and transparency for review. Any non-trivial branch effort should
have a merge strategy for getting it reviewed, tested and merged.


On Tue, Mar 22, 2016 at 9:46 PM, Gangumalla, Uma 
wrote:

> > is it possible for me to setup a branch, self review+commit to that
> >branch, then request a branch merge?
> Basically this is something like Commit-Then-Review(here review later)
> process right. I have not seen we followed this approach here( not sure if
> I missed some branches followed that). Even though original author code
> quality is good, there would always be chances for missing somethings. So,
> peer review is important because the other eye can catch some issues which
> original author might overlooked (general review advantage :-)). In this
> case for branch merge we need three  +1s. if we face difficult in getting
> one +1, then I am afraid that we may face more difficult to get reviewers
> a the time of merge, because code base much larger than normal patch. Some
> times we suggest contributors to split patches into multiple JIRAs of
> patch size is becoming larger. It is better to find some reviewers for the
> branch and creating branch could turn into healthy merge later.
>
> Colin suggestion sounds good to me. How about providing more details and
> find some reviewers (who is more familiar in that area etc)?
>
> If this is general question question for branch policy MY answer is ³no²
> for "self review+commit to that branch, then request a branch merge². But
> for this kind of special cases where we are sure that we may not have
> enough reviewers for branch, having dev mailing discussion about that
> JIRA/branch proposal and see how to go about that changes may be good idea
> instead of going ahead finishing work and raising merge vote thread ?
> (Something like this what you did now :-))
>
> Just my thoughts on this discussion.
>
> Thanks & Regards,
> Uma
>
> On 3/22/16, 9:14 AM, "Allen Wittenauer"  wrote:
>
> >Since it¹s nearly impossible for me to get timely reviews for some build
> >and script changes, is it possible for me to setup a branch, self
> >review+commit to that branch, then request a branch merge?
>
>

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1231

2016-03-22 Thread Apache Jenkins Server 
See 

Changes:

[cmccabe] HDFS-10189. PacketResponder#toString should include the downstreams 
for

--
[...truncated 3846 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.149 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.418 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating

Re: Branch policy question

2016-03-22 Thread Gangumalla, Uma 
> is it possible for me to setup a branch, self review+commit to that
>branch, then request a branch merge?
Basically this is something like Commit-Then-Review(here review later)
process right. I have not seen we followed this approach here( not sure if
I missed some branches followed that). Even though original author code
quality is good, there would always be chances for missing somethings. So,
peer review is important because the other eye can catch some issues which
original author might overlooked (general review advantage :-)). In this
case for branch merge we need three  +1s. if we face difficult in getting
one +1, then I am afraid that we may face more difficult to get reviewers
a the time of merge, because code base much larger than normal patch. Some
times we suggest contributors to split patches into multiple JIRAs of
patch size is becoming larger. It is better to find some reviewers for the
branch and creating branch could turn into healthy merge later.

Colin suggestion sounds good to me. How about providing more details and
find some reviewers (who is more familiar in that area etc)?

If this is general question question for branch policy MY answer is ³no²
for "self review+commit to that branch, then request a branch merge². But
for this kind of special cases where we are sure that we may not have
enough reviewers for branch, having dev mailing discussion about that
JIRA/branch proposal and see how to go about that changes may be good idea
instead of going ahead finishing work and raising merge vote thread ?
(Something like this what you did now :-))

Just my thoughts on this discussion.

Thanks & Regards,
Uma

On 3/22/16, 9:14 AM, "Allen Wittenauer"  wrote:

>Since it¹s nearly impossible for me to get timely reviews for some build
>and script changes, is it possible for me to setup a branch, self
>review+commit to that branch, then request a branch merge?

[jira] [Created] (HADOOP-12955) checknative failed when checking ISA-L library

2016-03-22 Thread Kai Zheng (JIRA) 
Kai Zheng created HADOOP-12955:
--

 Summary: checknative failed when checking ISA-L library
 Key: HADOOP-12955
 URL: https://issues.apache.org/jira/browse/HADOOP-12955
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Kai Zheng
Assignee: Kai Zheng


Ref. the comment 
[here|https://issues.apache.org/jira/browse/HADOOP-11540?focusedCommentId=15207619=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207619].
 

When run hadoop checknative, it also failed. Got something like below from log:
{noformat}
Stack: [0x7f2b9d405000,0x7f2b9d506000],  sp=0x7f2b9d504748,  free 
space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa90c90]  UTF8::unicode_length(char const*)+0x0
V  [libjvm.so+0x6ddfc3]  jni_NewStringUTF+0xc3
j  
org.apache.hadoop.io.erasurecode.ErasureCodeNative.getLibraryName()Ljava/lang/String;+0
j  org.apache.hadoop.util.NativeLibraryChecker.main([Ljava/lang/String;)V+212
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68c616]  JavaCalls::call_helper(JavaValue*, methodHandle*, 
JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6cdc32]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, 
JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x362
V  [libjvm.so+0x6ea63a]  jni_CallStaticVoidMethod+0x17a
C  [libjli.so+0x7bcc]  JavaMain+0x80c
C  [libpthread.so.0+0x8182]  start_thread+0xc2

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  
org.apache.hadoop.io.erasurecode.ErasureCodeNative.getLibraryName()Ljava/lang/String;+0
j  org.apache.hadoop.util.NativeLibraryChecker.main([Ljava/lang/String;)V+212
v  ~StubRoutines::call_stub
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-Common-trunk #2522

2016-03-22 Thread Apache Jenkins Server 
See

Re: Branch policy question

2016-03-22 Thread Colin McCabe 
If the underlying problem is lack of reviewers for these improvements,
how about a design doc giving some motivation for the improvements and
explaining how they'll be implemented?  Then we can decide if a branch
or a few JIRAs on trunk makes more sense.

The description for HADOOP-12857 is just "let's rework this to be
smarter" which could be the description of any JIRA in Hadoop (except
for the ones which make things dumber, of course).

best,
Colin


On Tue, Mar 22, 2016, at 11:23, Andrew Wang wrote:
> A branch sounds fine, but how are we going to get 3 +1's to merge it? If
> it's hard to find one reviewer, seems even harder to find two.
> 
> On Tue, Mar 22, 2016 at 10:56 AM, Allen Wittenauer <
> allenwittena...@yahoo.com.invalid> wrote:
> 
> >
> > > On Mar 22, 2016, at 10:49 AM, larry mccay  wrote:
> > >
> > > That sounds like a reasonable approach and valid use of branches to me.
> > >
> > > Perhaps a set of functional tests could be provided/identified that would
> > > help the review process by showing backward compatibility along with new
> > > extensions for things like dynamic commands?
> > >
> >
> > This is going into trunk, so no need for backward compatibility.
> >

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1230

2016-03-22 Thread Apache Jenkins Server 
See 

Changes:

[gtcarrera9] MAPREDUCE-6110. JobHistoryServer CLI throws NullPointerException 
with

--
[...truncated 3847 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.556 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.628 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating

[jira] [Created] (HADOOP-12954) Add a way to change hadoop.security.token.service.use_ip

2016-03-22 Thread Robert Kanter (JIRA) 
Robert Kanter created HADOOP-12954:
--

 Summary: Add a way to change hadoop.security.token.service.use_ip
 Key: HADOOP-12954
 URL: https://issues.apache.org/jira/browse/HADOOP-12954
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Robert Kanter
Assignee: Robert Kanter


Currently, {{hadoop.security.token.service.use_ip}} is set on JVM startup via:
{code:java}
  static {
Configuration conf = new Configuration();
boolean useIp = conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP,
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP_DEFAULT);
setTokenServiceUseIp(useIp);
  }
{code}

This is a problem for clients, such as Oozie, who don't add *-site.xml files to 
their classpath.  Oozie normally creates a {{JobClient}} and passes a 
{{Configuration}} to it with the proper configs we need.  However, because 
{{hadoop.security.token.service.use_ip}} is specified in a static block like 
this, and there's no API to change it, Oozie has no way to set it to the 
non-default value.

I propose we add a {{setConfiguration}} method which takes a {{Configuration}} 
and rereads {{hadoop.security.token.service.use_ip}}.  There's a few other 
properties that are also loaded statically on startup that can be reloaded here 
as well.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Accept Chimera as new Apache Commons Component

2016-03-22 Thread Chris Nauroth 
+1 (non-binding)

--Chris Nauroth




On 3/21/16, 1:45 AM, "Benedikt Ritter"  wrote:

>Hi all,
>
>after long discussions I think we have gathered enough information to
>decide whether we want to accept the Chimera project as a new Apache
>Commons component.
>
>Proposed name: Apache Commons Crypto
>Proposal text:
>https://github.com/intel-hadoop/chimera/blob/master/PROPOSAL.html
>Initial Code Base:  https://github.com/intel-hadoop/chimera/
>Initial Committers (Names in alphabetical order):
>- Aaron T. Myers (a...@apache.org, Apache Hadoop PMC, one of the original
>Crypto dev team in Apache Hadoop)
>- Andrew Wang (w...@apache.org, Apache Hadoop PMC, one of the original
>Crypto dev team in Apache Hadoop)
>- Chris Nauroth (cnaur...@apache.org, Apache Hadoop PMC and active
>reviewer)
>- Colin P. McCabe (cmcc...@apache.org, Apache Hadoop PMC, one of the
>original Crypto dev team in Apache Hadoop)
>- Dapeng Sun (s...@apache.org, Apache Sentry Committer, Chimera
>contributor)
>- Dian Fu (dia...@apache.org, Apache Sqoop Committer, Chimera contributor)
>- Dong Chen (do...@apache.org, Apache Hive Committer,interested on
>Chimera)
>- Ferdinand Xu (x...@apache.org, Apache Hive Committer, Chimera
>contributor)
>- Haifeng Chen (haifengc...@apache.org, Chimera lead and code contributor)
>- Marcelo Vanzin (Apache Spark Committer, Chimera contributor)
>- Uma Maheswara Rao G (umamah...@apache.org, Apache Hadoop PMC, One of the
>original Crypto dev/review team in Apache Hadoop)
>- Yi Liu (y...@apache.org, Apache Hadoop PMC, One of the original Crypto
>dev/review team in Apache Hadoop)
>
>Please review the proposal and vote.
>This vote will close no sooner than 72 hours from now, i.e. after 0900
>GMT 24-Mar 2016
>
>  [ ] +1 Accept Chimera as new Apache Commons Component
>  [ ] +0 OK, but...
>  [ ] -0 OK, but really should fix...
>  [ ] -1 I oppose this because...
>
>Thank you!
>Benedikt

[jira] [Created] (HADOOP-12953) New API for libhdfs to get FileSystem object as a proxy user

2016-03-22 Thread Uday Kale (JIRA) 
Uday Kale created HADOOP-12953:
--

 Summary: New API for libhdfs to get FileSystem object as a proxy 
user
 Key: HADOOP-12953
 URL: https://issues.apache.org/jira/browse/HADOOP-12953
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Uday Kale
Assignee: Uday Kale






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Branch policy question

2016-03-22 Thread Sean Busbey 
VOTE threads tend to get more eyes than random JIRAs.

On Tue, Mar 22, 2016 at 1:23 PM, Andrew Wang 
wrote:

> A branch sounds fine, but how are we going to get 3 +1's to merge it? If
> it's hard to find one reviewer, seems even harder to find two.
>
> On Tue, Mar 22, 2016 at 10:56 AM, Allen Wittenauer <
> allenwittena...@yahoo.com.invalid> wrote:
>
> >
> > > On Mar 22, 2016, at 10:49 AM, larry mccay 
> wrote:
> > >
> > > That sounds like a reasonable approach and valid use of branches to me.
> > >
> > > Perhaps a set of functional tests could be provided/identified that
> would
> > > help the review process by showing backward compatibility along with
> new
> > > extensions for things like dynamic commands?
> > >
> >
> > This is going into trunk, so no need for backward compatibility.
> >
>



-- 
busbey

Re: Branch policy question

2016-03-22 Thread Andrew Wang 
A branch sounds fine, but how are we going to get 3 +1's to merge it? If
it's hard to find one reviewer, seems even harder to find two.

On Tue, Mar 22, 2016 at 10:56 AM, Allen Wittenauer <
allenwittena...@yahoo.com.invalid> wrote:

>
> > On Mar 22, 2016, at 10:49 AM, larry mccay  wrote:
> >
> > That sounds like a reasonable approach and valid use of branches to me.
> >
> > Perhaps a set of functional tests could be provided/identified that would
> > help the review process by showing backward compatibility along with new
> > extensions for things like dynamic commands?
> >
>
> This is going into trunk, so no need for backward compatibility.
>

Re: Branch policy question

2016-03-22 Thread Allen Wittenauer 

> On Mar 22, 2016, at 10:49 AM, larry mccay  wrote:
> 
> That sounds like a reasonable approach and valid use of branches to me.
> 
> Perhaps a set of functional tests could be provided/identified that would
> help the review process by showing backward compatibility along with new
> extensions for things like dynamic commands?
> 

This is going into trunk, so no need for backward compatibility.

Re: Branch policy question

2016-03-22 Thread larry mccay 
That sounds like a reasonable approach and valid use of branches to me.

Perhaps a set of functional tests could be provided/identified that would
help the review process by showing backward compatibility along with new
extensions for things like dynamic commands?


On Tue, Mar 22, 2016 at 12:14 PM, Allen Wittenauer  wrote:

>
> Since it’s nearly impossible for me to get timely reviews for some
> build and script changes, is it possible for me to setup a branch, self
> review+commit to that branch, then request a branch merge?  I’m basically
> looking at doing this for HADOOP-12857 + HADOOP-12930 and their subtasks
> esp since they are very order dependent.  I don’t know if a branch is
> easier to review, honestly, but at least I wouldn’t be blocked on making
> progress...
>
> Thanks.

Branch policy question

2016-03-22 Thread Allen Wittenauer 

Since it’s nearly impossible for me to get timely reviews for some 
build and script changes, is it possible for me to setup a branch, self 
review+commit to that branch, then request a branch merge?  I’m basically 
looking at doing this for HADOOP-12857 + HADOOP-12930 and their subtasks esp 
since they are very order dependent.  I don’t know if a branch is easier to 
review, honestly, but at least I wouldn’t be blocked on making progress...

Thanks.

[jira] [Created] (HADOOP-12952) /BUILDING example of zero-docs dist should skip javadocs

2016-03-22 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-12952:
---

 Summary: /BUILDING example of zero-docs dist should skip javadocs
 Key: HADOOP-12952
 URL: https://issues.apache.org/jira/browse/HADOOP-12952
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, documentation
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran


The examples for building distributions include how to create one without any 
documentation. But it includes the javadoc stage in the build, which is very 
slow.

Adding {{-Dmaven.javadoc.skip=true}} skips that phase, and helps round out the 
parameters to a build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)