[jira] [Commented] (MAPREDUCE-6110) JobHistoryServer CLI throws NullPointerException with job ids that do not exist

2016-03-21 Thread Li Lu (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205398#comment-15205398
 ] 

Li Lu commented on MAPREDUCE-6110:
--

LGTM. +1. Will commit after about a day. 

> JobHistoryServer CLI throws NullPointerException with job ids that do not 
> exist
> ---
>
> Key: MAPREDUCE-6110
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6110
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Reporter: Li Lu
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6110.01.patch, MAPREDUCE-6110.02.patch, 
> MAPREDUCE-6110.03.patch
>
>
> When using JobHistoryServer CLI to query a job id that does not exist on the 
> server, it may throw NullPointerException sometimes. 
> I tried "mapred job -events  0 100", and the result was:
> Exception in thread "main" java.lang.NullPointerException
>   at org.apache.hadoop.mapreduce.tools.CLI.listEvents(CLI.java:487)
>   at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:316)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>   at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1237)
> Similar symptoms also appear with -list-attempt-ids, but were fine with 
> -status and -set-priority. 
> I traced back to CLI.listEvents, and line 487 is:
> {code}
> TaskCompletionEvent[] events = job.
>   getTaskCompletionEvents(fromEventId, numEvents);
> {code}
> The job object is obtained from JobID.forName(jobid)) (line 316), which will 
> return null if the job does not exist on server. 
> Maybe we want to have some behaviors consistent with -status here, by simply 
> reporting jobId does not exist? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6580) Test failure : TestMRJobsWithProfiler

2016-03-21 Thread Eric Badger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated MAPREDUCE-6580:
---
Status: Patch Available  (was: Open)

> Test failure : TestMRJobsWithProfiler
> -
>
> Key: MAPREDUCE-6580
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6580
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6580.001.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestMRJobsWithProfiler fails intermittently
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) 
>  Time elapsed: 133.116 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6580) Test failure : TestMRJobsWithProfiler

2016-03-21 Thread Eric Badger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated MAPREDUCE-6580:
---
Attachment: MAPREDUCE-6580.001.patch

> Test failure : TestMRJobsWithProfiler
> -
>
> Key: MAPREDUCE-6580
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6580
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6580.001.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestMRJobsWithProfiler fails intermittently
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) 
>  Time elapsed: 133.116 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

2016-03-21 Thread Sangjin Lee (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee resolved MAPREDUCE-6372.

   Resolution: Resolved
Fix Version/s: YARN-2928

I have addressed the known issues as part of MAPREDUCE-6546.

> clean up several issues with TimelineServicePerformance
> ---
>
> Key: MAPREDUCE-6372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
> Fix For: YARN-2928
>
>
> We found a few issues with the TimelineServicePerformanceV2 test driver while 
> running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory

2016-03-21 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205315#comment-15205315
 ] 

Hadoop QA commented on MAPREDUCE-6315:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s 
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 10s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch 
generated 1 new + 521 unchanged - 0 fixed = 522 total (was 521) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 32s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 12s {color} 
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK 
v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 51s 
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed 
with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 28s {color} 
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK 
v1.7

[jira] [Assigned] (MAPREDUCE-6580) Test failure : TestMRJobsWithProfiler

2016-03-21 Thread Eric Badger (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger reassigned MAPREDUCE-6580:
--

Assignee: Eric Badger

> Test failure : TestMRJobsWithProfiler
> -
>
> Key: MAPREDUCE-6580
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6580
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestMRJobsWithProfiler fails intermittently
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) 
>  Time elapsed: 133.116 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6580) Test failure : TestMRJobsWithProfiler

2016-03-21 Thread Eric Badger (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205113#comment-15205113
 ] 

Eric Badger commented on MAPREDUCE-6580:


I am able to get TestMRJobsWithProfiler#testDifferentProfilers to fail 
consistently on my Mac. However, I am not able to get it to fail on my Linux 
box or on either box when running testDefaultProfiler. On my mac, I have found 
that the problem is with the hprof option "times" in 
TestMRJobsWithProfiler.java:137 
(-agentlib:hprof=cpu=times,heap=sites,force=n,thread=y,verbose=n," + "file=%s). 
This is causing the JVM to screw up and fail seemingly before it even begins to 
start the container. 

Additionally, the "times" option is incredibly expensive and slows the JVM down 
a ton, making the test very likely to timeout on a busy Jenkins instance given 
the 150 second test timeout. IMO using "samples" instead of "times" would be 
much better. Using samples, the test passes on my mac and speeds the test up by 
almost 2x (74.8s vs. 145.5s on my Linux machine). Using "samples" would get rid 
of the hprof error on my mac and also speed up the test considerably, reducing 
the chance of test timeouts.

Is there any reason that we can't use "samples" instead of "times" for the 
profiler? 

> Test failure : TestMRJobsWithProfiler
> -
>
> Key: MAPREDUCE-6580
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6580
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestMRJobsWithProfiler fails intermittently
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
> testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler) 
>  Time elapsed: 133.116 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<4> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory

2016-03-21 Thread Gera Shegalov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gera Shegalov updated MAPREDUCE-6315:
-
Attachment: MAPREDUCE-6315.002.patch

Ran into an AM crash again, and found this AM log recovery really useful. So 
rebasing and fixing the test failures (flush count because I am making 
AM_STARTED forcelogged).  

> Implement retrieval of logs for crashed MR-AM via jhist in the staging 
> directory
> 
>
> Key: MAPREDUCE-6315
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client, mr-am
>Affects Versions: 2.7.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Critical
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch
>
>
> When all AM attempts crash, there is no record of them in JHS. Thus no easy 
> way to get the logs. This JIRA automates the procedure by utilizing the jhist 
> file in the staging directory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6655) Fix a typo (STRICT_IE6) in Encrypted Shuffle

2016-03-21 Thread Wei-Chiu Chuang (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204202#comment-15204202
 ] 

Wei-Chiu Chuang commented on MAPREDUCE-6655:


The asf licensing warning  and unit test error are unrelated to this patch, as 
this patch only modified documentation and the description of a 
core-default.xml property.

> Fix a typo (STRICT_IE6) in Encrypted Shuffle
> 
>
> Key: MAPREDUCE-6655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6655
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
>  Labels: typo
> Attachments: MAPREDUCE-6655.001.patch
>
>
> Found a typo in Encrypted Shuffle documentation (STRICT_I6 --> STRICT_IE6).
> The typo also appears in core-default.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)