Hadoop-Mapreduce-trunk-Java8 - Build # 1388 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/1388/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9269 lines...]
  TestHsWebServicesAcls.testGetJobConfAcls:138 » NoClassDefFound Could not 
initi...
  TestHsWebServicesAcls.testGetJobTaskAcls:172 » NoClassDefFound Could not 
initi...
  TestHsWebServicesAcls.testGetJobTasksAcls:155 » NoClassDefFound Could not 
init...
  TestHsWebServicesAcls.testGetJobTaskAttemptIdCountersAcls:242 » 
NoClassDefFound
  TestJobHistoryEntities.testCopmletedJobReportWithZeroTasks:125 » 
NoClassDefFound
  TestJobHistoryEntities.testGetTaskAttemptCompletionEvent:210 » NoClassDefFound
  TestJobHistoryEntities.testCompletedTaskAttempt:171 » NoClassDefFound 
org/apac...
  TestJobHistoryEntities.testCompletedJob:99 » NoClassDefFound 
org/apache/hadoop...
  TestJobHistoryEntities.testCompletedJobWithDiagnostics:264 » NoClassDefFound 
o...
  TestJobHistoryEntities.testCompletedTask:142 » NoClassDefFound 
org/apache/hado...
  TestJobHistoryEntities.testCopmletedJobReportWithZeroTasks:125 » 
NoClassDefFound
  TestJobHistoryEntities.testGetTaskAttemptCompletionEvent:210 » NoClassDefFound
  TestJobHistoryEntities.testCompletedTaskAttempt:171 » NoClassDefFound 
org/apac...
  TestJobHistoryEntities.testCompletedJob:99 » NoClassDefFound 
org/apache/hadoop...
  TestJobHistoryEntities.testCompletedJobWithDiagnostics:264 » NoClassDefFound 
o...
  TestJobHistoryEntities.testCompletedTask:142 » NoClassDefFound 
org/apache/hado...

Tests run: 207, Failures: 8, Errors: 21, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.412 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:38 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 25.213 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.639 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [10:32 min]
[INFO] Apache Hadoop MapReduce HistoryServer . FAILURE [06:05 min]
[INFO] Apache Hadoop MapReduce JobClient . SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 18:49 min
[INFO] Finished at: 2016-05-12T03:15:11+00:00
[INFO] Final Memory: 38M/235M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-hs: ExecutionException: 
java.lang.RuntimeException: The forked VM terminated without properly saying 
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
 && /home/jenkins/tools/java/jdk1.8.0/jre/bin/java -Xmx2048m 
-XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/target/surefire/surefirebooter6451416354686986110.jar
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/target/surefire/surefire3460155293781293656tmp
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/target/surefire/surefire_133938803200588785492tmp
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-hs
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

Hadoop-Mapreduce-trunk - Build # 3307 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/3307/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32738 lines...]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ 
hadoop-mapreduce ---
[INFO] Installing 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-mapreduce/3.0.0-SNAPSHOT/hadoop-mapreduce-3.0.0-SNAPSHOT.pom
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  4.447 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:53 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 30.379 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  5.534 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [08:37 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [05:50 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:47 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SUCCESS [  1.503 s]
[INFO] Apache Hadoop MapReduce NativeTask  SUCCESS [09:00 min]
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SUCCESS [  0.259 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:13 h
[INFO] Finished at: 2016-05-12T02:16:16+00:00
[INFO] Final Memory: 37M/595M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
3 tests failed.
FAILED:  org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort

Error Message:
Output directory not empty expected:<0> but was:<4>

Stack Trace:
java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)


FAILED:  org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv

Error Message:
The environment checker job failed.

Stack Trace:
java.lang.AssertionError: The environment checker job failed.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
at 
org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv(TestMiniMRChildTask.java:472)


FAILED:  org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskOldEnv

Error

Hadoop-Mapreduce-trunk-Java8 - Build # 1387 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/1387/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9541 lines...]
Running org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.506 sec - in 
org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.TestMapCollection
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.673 sec - 
in org.apache.hadoop.mapreduce.TestMapCollection
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileMergeProgress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.667 sec - in 
org.apache.hadoop.io.TestSequenceFileMergeProgress

Results :

Failed tests: 
  TestMiniMRChildTask.testTaskEnv:472->runTestTaskEnv:550 The environment 
checker job failed.
  TestMiniMRChildTask.testTaskOldEnv:496->runTestTaskEnv:550 The environment 
checker job failed.
  TestClusterMRNotification>NotificationTestCase.testMR:205 expected:<6> but 
was:<4>

Tests run: 533, Failures: 3, Errors: 0, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.444 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:40 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 25.677 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.312 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [10:12 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [06:24 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:47 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:06 h
[INFO] Finished at: 2016-05-12T01:17:57+00:00
[INFO] Final Memory: 34M/144M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
3 tests failed.
FAILED:  org.apache.hadoop.mapred.TestClusterMRNotification.testMR

Error Message:
expected:<6> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<6> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Jian He 
For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
for 2.8

MAPREDUCE-6288
YARN-1815
YARN-4685
YARN-4844

The rest are either improvements or long-standing issues and does not qualify 
release blocker, IMO.
I think we’ll try to get these 4 jiras in asap. The rest will be on best 
effort, resolve as much as possible and move them out if not resolved in time.

Jian

On May 11, 2016, at 5:37 PM, Wangda Tan 
> wrote:

Sounds good to me :).

Jian and I have looked at all existing 2.8.0 blockers and criticals today.
To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
out. Left comments on these JIRAs asked original owners, plan to update
target version of these JIRAs early next week.

Will keep this thread updated.

Thanks,
Wangda


On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> wrote:

How about this? I'll review the HADOOP/HDFS bugs in that list to come up
with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
report the list here. Then folks can chime in if you agree

Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?

Thanks,
Sangjin

On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> wrote:

+1, we should close such staled JIRAs to avoid doing unnecessary checks
for
every releases.

I'm working on reviewing YARN/MR critical/blocker patches currently, it
gonna very helpful if someone else can help with reviewing Common/HDFS
JIRAs.

Thanks,
Wangda


On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> wrote:

Where do we stand in terms of closing out blocker/critical issues for
2.8.0? I still see 50 open JIRAs in Vinod's list:
https://issues.apache.org/jira/issues/?filter=12334985

But I see a lot of JIRAs with no patches or very stale patches. It
would be
a good exercise to come up with the list of JIRAs that we need to block
2.8.0 for and focus our attention on closing them out. Thoughts?

Thanks,
Sangjin

On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 


wrote:


On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
vino...@apache.org>
wrote:

We are not converging - there’s still 58 more. I need help from the
community in addressing / review 2.8.0 blockers. If folks can start
with
reviewing Patch available tickets, that’ll be great.




I'm still doing the s3a stuff, other people testing and reviewing this
stuff welcome.

in particular, I could do with others playing with this patch of mine,
which adds counters and things into S3a, based on the azure
instrumentation

https://issues.apache.org/jira/browse/HADOOP-13028

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Sangjin Lee 
I see the following list of JIRAs from HADOOP and HDFS as blockers for
2.8.0. Other JIRAs are either old issues with little movement or new issues
that don't appear to be as critical/ready.

- HADOOP-12893
- HADOOP-12892
- HADOOP-10940 (patch ready?)
- HADOOP-12971 (can be done relatively quickly?)
- HDFS-7959
- HDFS-7597 (needs review/some more work?)

I would propose moving the rest out of scope for 2.8.0 (23 JIRAs). Let me
know what you think.


On Wed, May 11, 2016 at 5:37 PM, Wangda Tan  wrote:

> Sounds good to me :).
>
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
>
> Will keep this thread updated.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee  wrote:
>
>> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
>> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
>> report the list here. Then folks can chime in if you agree
>>
>> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>>
>> Thanks,
>> Sangjin
>>
>> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan  wrote:
>>
>>> +1, we should close such staled JIRAs to avoid doing unnecessary checks
>>> for
>>> every releases.
>>>
>>> I'm working on reviewing YARN/MR critical/blocker patches currently, it
>>> gonna very helpful if someone else can help with reviewing Common/HDFS
>>> JIRAs.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee  wrote:
>>>
>>> > Where do we stand in terms of closing out blocker/critical issues for
>>> > 2.8.0? I still see 50 open JIRAs in Vinod's list:
>>> > https://issues.apache.org/jira/issues/?filter=12334985
>>> >
>>> > But I see a lot of JIRAs with no patches or very stale patches. It
>>> would be
>>> > a good exercise to come up with the list of JIRAs that we need to block
>>> > 2.8.0 for and focus our attention on closing them out. Thoughts?
>>> >
>>> > Thanks,
>>> > Sangjin
>>> >
>>> > On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran <
>>> ste...@hortonworks.com>
>>> > wrote:
>>> >
>>> > >
>>> > > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
>>> vino...@apache.org>
>>> > > wrote:
>>> > > >
>>> > > > We are not converging - there’s still 58 more. I need help from the
>>> > > community in addressing / review 2.8.0 blockers. If folks can start
>>> with
>>> > > reviewing Patch available tickets, that’ll be great.
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > I'm still doing the s3a stuff, other people testing and reviewing
>>> this
>>> > > stuff welcome.
>>> > >
>>> > > in particular, I could do with others playing with this patch of
>>> mine,
>>> > > which adds counters and things into S3a, based on the azure
>>> > instrumentation
>>> > >
>>> > > https://issues.apache.org/jira/browse/HADOOP-13028
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Wangda Tan 
Sounds good to me :).

Jian and I have looked at all existing 2.8.0 blockers and criticals today.
To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
out. Left comments on these JIRAs asked original owners, plan to update
target version of these JIRAs early next week.

Will keep this thread updated.

Thanks,
Wangda


On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee  wrote:

> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
>
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
>
> Thanks,
> Sangjin
>
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan  wrote:
>
>> +1, we should close such staled JIRAs to avoid doing unnecessary checks
>> for
>> every releases.
>>
>> I'm working on reviewing YARN/MR critical/blocker patches currently, it
>> gonna very helpful if someone else can help with reviewing Common/HDFS
>> JIRAs.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee  wrote:
>>
>> > Where do we stand in terms of closing out blocker/critical issues for
>> > 2.8.0? I still see 50 open JIRAs in Vinod's list:
>> > https://issues.apache.org/jira/issues/?filter=12334985
>> >
>> > But I see a lot of JIRAs with no patches or very stale patches. It
>> would be
>> > a good exercise to come up with the list of JIRAs that we need to block
>> > 2.8.0 for and focus our attention on closing them out. Thoughts?
>> >
>> > Thanks,
>> > Sangjin
>> >
>> > On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran > >
>> > wrote:
>> >
>> > >
>> > > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
>> vino...@apache.org>
>> > > wrote:
>> > > >
>> > > > We are not converging - there’s still 58 more. I need help from the
>> > > community in addressing / review 2.8.0 blockers. If folks can start
>> with
>> > > reviewing Patch available tickets, that’ll be great.
>> > > >
>> > > >
>> > >
>> > >
>> > > I'm still doing the s3a stuff, other people testing and reviewing this
>> > > stuff welcome.
>> > >
>> > > in particular, I could do with others playing with this patch of mine,
>> > > which adds counters and things into S3a, based on the azure
>> > instrumentation
>> > >
>> > > https://issues.apache.org/jira/browse/HADOOP-13028
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Sangjin Lee 
How about this? I'll review the HADOOP/HDFS bugs in that list to come up
with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
report the list here. Then folks can chime in if you agree

Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?

Thanks,
Sangjin

On Wed, May 11, 2016 at 4:26 PM, Wangda Tan  wrote:

> +1, we should close such staled JIRAs to avoid doing unnecessary checks for
> every releases.
>
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
>
> Thanks,
> Wangda
>
>
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee  wrote:
>
> > Where do we stand in terms of closing out blocker/critical issues for
> > 2.8.0? I still see 50 open JIRAs in Vinod's list:
> > https://issues.apache.org/jira/issues/?filter=12334985
> >
> > But I see a lot of JIRAs with no patches or very stale patches. It would
> be
> > a good exercise to come up with the list of JIRAs that we need to block
> > 2.8.0 for and focus our attention on closing them out. Thoughts?
> >
> > Thanks,
> > Sangjin
> >
> > On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> > wrote:
> >
> > >
> > > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> > > wrote:
> > > >
> > > > We are not converging - there’s still 58 more. I need help from the
> > > community in addressing / review 2.8.0 blockers. If folks can start
> with
> > > reviewing Patch available tickets, that’ll be great.
> > > >
> > > >
> > >
> > >
> > > I'm still doing the s3a stuff, other people testing and reviewing this
> > > stuff welcome.
> > >
> > > in particular, I could do with others playing with this patch of mine,
> > > which adds counters and things into S3a, based on the azure
> > instrumentation
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-13028
> > >
> > >
> > >
> >
>

[jira] [Resolved] (MAPREDUCE-6099) Adding getSplits(JobContext job, List stats) to mapreduce CombineFileInputFormat

2016-05-11 Thread Jian He (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He resolved MAPREDUCE-6099.

Resolution: Won't Fix

Close as Jason mentioned 

> Adding  getSplits(JobContext job, List stats) to mapreduce 
> CombineFileInputFormat
> -
>
> Key: MAPREDUCE-6099
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6099
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 2.4.1
>Reporter: Pankit Thapar
>Priority: Critical
> Attachments: MAPREDUCE-6099.patch
>
>
> Currently we have getSplits(JobContext job) in CombineFileInputFormat. 
> This api does not give freedom to the client to create a list if file status 
> it self and then create splits on the resultant List stats.
> The client might be able to perform some filtering on its end on the File 
> sets in the input paths. For the reasons, above it would be a good idea to 
> have getSplits(JobContext, List).
> Please let me know what you think about this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Wangda Tan 
+1, we should close such staled JIRAs to avoid doing unnecessary checks for
every releases.

I'm working on reviewing YARN/MR critical/blocker patches currently, it
gonna very helpful if someone else can help with reviewing Common/HDFS
JIRAs.

Thanks,
Wangda


On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee  wrote:

> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
>
> But I see a lot of JIRAs with no patches or very stale patches. It would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
>
> Thanks,
> Sangjin
>
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> wrote:
>
> >
> > > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli 
> > wrote:
> > >
> > > We are not converging - there’s still 58 more. I need help from the
> > community in addressing / review 2.8.0 blockers. If folks can start with
> > reviewing Patch available tickets, that’ll be great.
> > >
> > >
> >
> >
> > I'm still doing the s3a stuff, other people testing and reviewing this
> > stuff welcome.
> >
> > in particular, I could do with others playing with this patch of mine,
> > which adds counters and things into S3a, based on the azure
> instrumentation
> >
> > https://issues.apache.org/jira/browse/HADOOP-13028
> >
> >
> >
>

Re: [Release thread] 2.8.0 release activities

2016-05-11 Thread Sangjin Lee 
Where do we stand in terms of closing out blocker/critical issues for
2.8.0? I still see 50 open JIRAs in Vinod's list:
https://issues.apache.org/jira/issues/?filter=12334985

But I see a lot of JIRAs with no patches or very stale patches. It would be
a good exercise to come up with the list of JIRAs that we need to block
2.8.0 for and focus our attention on closing them out. Thoughts?

Thanks,
Sangjin

On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
wrote:

>
> > On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli 
> wrote:
> >
> > We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start with
> reviewing Patch available tickets, that’ll be great.
> >
> >
>
>
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
>
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure instrumentation
>
> https://issues.apache.org/jira/browse/HADOOP-13028
>
>
>

Hadoop-Mapreduce-trunk-Java8 - Build # 1386 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/1386/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 8921 lines...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.64 sec - in 
org.apache.hadoop.mapreduce.v2.hs.TestJobHistory
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.047 sec - in 
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.v2.hs.TestHistoryFileManager
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.802 sec - in 
org.apache.hadoop.mapreduce.v2.hs.TestHistoryFileManager
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.v2.hs.TestJHSDelegationTokenSecretManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.621 sec - in 
org.apache.hadoop.mapreduce.v2.hs.TestJHSDelegationTokenSecretManager

Results :

Tests in error: 
  TestJobListCache.testAddExisting:39 »  test timed out after 1000 milliseconds

Tests run: 210, Failures: 0, Errors: 1, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  4.085 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [02:31 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 36.711 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  7.618 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [13:32 min]
[INFO] Apache Hadoop MapReduce HistoryServer . FAILURE [08:43 min]
[INFO] Apache Hadoop MapReduce JobClient . SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 25:37 min
[INFO] Finished at: 2016-05-11T20:40:11+00:00
[INFO] Final Memory: 38M/193M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-hs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-hs
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.hadoop.mapreduce.v2.hs.TestJobListCache.testAddExisting

Error Message:
test timed out after 1000 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 1000 milliseconds
at org.mockito.asm.Type.getDescriptor(Type.java:515)
at org.mockito.asm.Type.getDescriptor(Type.java:481)
at org.mockito.cglib.core.CodeEmitter.emit_field(CodeEmitter.java:469)
at org.mockito.cglib.core.CodeEmitter.getfield(CodeEmitter.java:426)
at 
org.mockito.cglib.proxy.MethodInterceptorGenerator$2.processCase(MethodInterceptorGenerator.java:226)
at org.mockito.cglib.core.EmitUtils$6.processCase(EmitUtils.java:294)
at

Hadoop-Mapreduce-trunk - Build # 3306 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/3306/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32405 lines...]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ 
hadoop-mapreduce ---
[INFO] Installing 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-mapreduce/3.0.0-SNAPSHOT/hadoop-mapreduce-3.0.0-SNAPSHOT.pom
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  4.006 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [02:21 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 35.601 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  7.376 s]
[INFO] Apache Hadoop MapReduce App ... FAILURE [09:55 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SKIPPED
[INFO] Apache Hadoop MapReduce JobClient . SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SUCCESS [  2.068 s]
[INFO] Apache Hadoop MapReduce NativeTask  SUCCESS [11:40 min]
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SUCCESS [  0.329 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 24:48 min
[INFO] Finished at: 2016-05-11T20:25:20+00:00
[INFO] Final Memory: 37M/827M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-app: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-app
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
16 tests failed.
FAILED:  
org.apache.hadoop.mapreduce.v2.app.job.impl.TestMapReduceChildJVM.testCommandLine

Error Message:
org/apache/hadoop/security/authentication/server/AuthenticationFilter

Stack Trace:
java.lang.NoClassDefFoundError: 
org/apache/hadoop/security/authentication/server/AuthenticationFilter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at 
org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:454)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:346)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:109)
at 
org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:291)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:276)
at

Hadoop-Mapreduce-trunk-Java8 - Build # 1385 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/1385/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 15097 lines...]
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.589 sec - in 
org.apache.hadoop.mapreduce.lib.chain.TestChainErrors
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.75 sec - in 
org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.mapreduce.TestMapCollection
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.215 sec - 
in org.apache.hadoop.mapreduce.TestMapCollection
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileMergeProgress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.417 sec - in 
org.apache.hadoop.io.TestSequenceFileMergeProgress

Results :

Failed tests: 
  TestMiniMRChildTask.testTaskEnv:472->runTestTaskEnv:550 The environment 
checker job failed.
  TestMiniMRChildTask.testTaskOldEnv:496->runTestTaskEnv:550 The environment 
checker job failed.

Tests run: 520, Failures: 2, Errors: 0, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.436 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:39 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 25.598 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.551 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [10:18 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [06:22 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:54 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:13 h
[INFO] Finished at: 2016-05-11T19:07:30+00:00
[INFO] Final Memory: 34M/146M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: ExecutionException: 
java.lang.RuntimeException: The forked VM terminated without properly saying 
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 && /home/jenkins/tools/java/jdk1.8.0/jre/bin/java -Xmx2048m 
-XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter3434796768488018636.jar
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire7152649698736072668tmp
 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_2606034661737822835944tmp
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

2016-05-11 Thread Junping Du 
bq. We'll need to be strict after the switch: all patches to go into branch 2 
will have to go through yetus as branch-2 patches, then cherry picked to trunk, 
or a separate patch on the same JIRA done for the branch-2. Assuming yetus 
still tests the branch-2 stuff on JDK7, that will check version compatibility 
pre-commit.
+1. That's something I want to say also. We should have separate process for 
backport effort between Trunk and branch-2 then. Otherwise, we could involve 
Java 7 related bug to branch-2 after the backport.

Thanks,

Junping

From: Steve Loughran 
Sent: Wednesday, May 11, 2016 5:20 PM
To: Akira AJISAKA
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

>
> On 10 May 2016, at 16:32, Akira AJISAKA  wrote:
>
> Hi developers,
>
> Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk.

+1

Given Robert Kanter first filed the patch to do this —why not give him the 
honour. In fact, why not have a webex screen share of the occasion so we can 
all celebrate?



> Given this is a critical change, I'm thinking we should get the consensus 
> first.
>
> One concern I think is, when the minimum version is set to JDK8, we need to 
> configure Jenkins to disable multi JDK test only in trunk.
>

LGTM

We'll need to be strict after the switch: all patches to go into branch 2 will 
have to go through yetus as branch-2 patches, then cherry picked to trunk, or a 
separate patch on the same JIRA done for the branch-2. Assuming yetus still 
tests the branch-2 stuff on JDK7, that will check version compatibility 
pre-commit
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

2016-05-11 Thread Steve Loughran 
> 
> On 10 May 2016, at 16:32, Akira AJISAKA  wrote:
> 
> Hi developers,
> 
> Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk.

+1

Given Robert Kanter first filed the patch to do this —why not give him the 
honour. In fact, why not have a webex screen share of the occasion so we can 
all celebrate?



> Given this is a critical change, I'm thinking we should get the consensus 
> first.
> 
> One concern I think is, when the minimum version is set to JDK8, we need to 
> configure Jenkins to disable multi JDK test only in trunk.
> 

LGTM

We'll need to be strict after the switch: all patches to go into branch 2 will 
have to go through yetus as branch-2 patches, then cherry picked to trunk, or a 
separate patch on the same JIRA done for the branch-2. Assuming yetus still 
tests the branch-2 stuff on JDK7, that will check version compatibility 
pre-commit
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Hadoop-Mapreduce-trunk-Java8 - Build # 1384 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/1384/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9513 lines...]
Running org.apache.hadoop.hdfs.TestNNBench
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.618 sec - in 
org.apache.hadoop.hdfs.TestNNBench
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileMergeProgress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.55 sec - in 
org.apache.hadoop.io.TestSequenceFileMergeProgress
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.conf.TestNoDefaultsJobConf
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.947 sec - in 
org.apache.hadoop.conf.TestNoDefaultsJobConf

Results :

Failed tests: 
  TestMRCJCFileOutputCommitter.testAbort:153 Output directory not empty 
expected:<0> but was:<4>
  TestMiniMRChildTask.testTaskEnv:472->runTestTaskEnv:550 The environment 
checker job failed.
  TestMiniMRChildTask.testTaskOldEnv:496->runTestTaskEnv:550 The environment 
checker job failed.

Tests run: 533, Failures: 3, Errors: 0, Skipped: 11

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  2.218 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [01:37 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 25.077 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  4.267 s]
[INFO] Apache Hadoop MapReduce App ... SUCCESS [10:04 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SUCCESS [06:21 min]
[INFO] Apache Hadoop MapReduce JobClient . FAILURE [  01:47 h]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:06 h
[INFO] Finished at: 2016-05-11T10:01:23+00:00
[INFO] Final Memory: 34M/142M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Java8/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-jobclient
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
3 tests failed.
FAILED:  org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort

Error Message:
Output directory not empty expected:<0> but was:<4>

Stack Trace:
java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)


FAILED:  org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv

Error Message:
The environment checker job failed.

Stack Trace:
java.lang.AssertionError: The environment checker job failed.
at org.junit.Assert.fail(Assert.java:88)
at

Hadoop-Mapreduce-trunk - Build # 3304 - Still Failing

2016-05-11 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/3304/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 32132 lines...]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-mapreduce ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-mapreduce ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @ 
hadoop-mapreduce ---
[INFO] Installing 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-mapreduce/3.0.0-SNAPSHOT/hadoop-mapreduce-3.0.0-SNAPSHOT.pom
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop MapReduce Client  SUCCESS [  3.708 s]
[INFO] Apache Hadoop MapReduce Core .. SUCCESS [02:22 min]
[INFO] Apache Hadoop MapReduce Common  SUCCESS [ 36.415 s]
[INFO] Apache Hadoop MapReduce Shuffle ... SUCCESS [  7.405 s]
[INFO] Apache Hadoop MapReduce App ... FAILURE [10:00 min]
[INFO] Apache Hadoop MapReduce HistoryServer . SKIPPED
[INFO] Apache Hadoop MapReduce JobClient . SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins . SUCCESS [  1.813 s]
[INFO] Apache Hadoop MapReduce NativeTask  SUCCESS [10:59 min]
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] Apache Hadoop MapReduce ... SUCCESS [  0.313 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 24:14 min
[INFO] Finished at: 2016-05-11T08:50:29+00:00
[INFO] Final Memory: 37M/828M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on 
project hadoop-mapreduce-client-app: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-mapreduce-client-app
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned

Error Message:
No Ta Started JH Event

Stack Trace:
java.lang.AssertionError: No Ta Started JH Event
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testTaskAttemptAssignedKilledHistory(TestTaskAttempt.java:403)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned(TestTaskAttempt.java:190)




-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

2016-05-11 Thread Sean Busbey 
+1 (non-binding)

reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.

tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.

On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer  wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) 
> to merge the HADOOP-12930 feature branch into trunk. This branch was 
> developed exclusively by me as per the discussion two months ago as a way to 
> make what would be a rather large patch hopefully easier to review.  The vast 
> majority of the branch is code movement in the same file, additional license 
> headers, maven assembly hooks for distribution, and variable renames. Not a 
> whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ 
> commands to allow for subcommands to be added or modified at runtime.  This 
> allows for individual users or entire sites to tweak the execution 
> environment to suit their local needs.  For example, it has been a practice 
> for some locations to change the distcp jar out for a custom one.  Using this 
> functionality, it is possible that the ‘hadoop distcp’ command could run the 
> local version without overwriting the bundled jar and for existing 
> documentation (read: results from Internet searches) to work as written 
> without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop 
> experience
> * operations teams that may be able to leverage existing 
> documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial 
> experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the 
> first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather 
> than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be 
> consistent with other bundled profiles, including the ones introduced in this 
> branch
>
> Documentation, including a ‘hello world’-style example, is in the 
> UnixShellGuide markdown file.  (Of course!)
>
>  I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks.  These are blocked by other 
> issues so that we may add unit testing to the shell code in those respective 
> areas.  I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org