[jira] [Resolved] (MAPREDUCE-6440) Duplicate Key in Json Output for Job details

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6440.
---
  Resolution: Duplicate
Target Version/s:   (was: )

This has been fixed by MAPREDUCE-7133.

> Duplicate Key in Json Output for Job details
> 
>
> Key: MAPREDUCE-6440
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6440
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Reporter: Anushri
>Priority: Minor
>
> Duplicate key in Json Output for Job details for the url : 
> http://:/ws/v1/history/mapreduce/jobs/job_id/tasks/task_id/attempts
> If the task type is "REDUCE" the json output for this url contains duplicate 
> key for "type".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6948) TestJobImpl.testUnusableNodeTransition failed

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6948.
---
Resolution: Cannot Reproduce

I agree as well.  I have not seen any recent precommit failures on 3.x releases 
for this unit test.

> TestJobImpl.testUnusableNodeTransition failed
> -
>
> Key: MAPREDUCE-6948
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6948
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Haibo Chen
>Assignee: Jim Brennan
>Priority: Major
>  Labels: unit-test
>
> *Error Message*
> expected: but was:
> *Stacktrace*
> java.lang.AssertionError: expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:1041)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:615)
> *Standard out*
> {code}
> 2017-08-30 10:12:21,928 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class 
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2017-08-30 10:12:21,939 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$StubbedJob
> 2017-08-30 10:12:21,940 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class 
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.jobhistory.EventType for class 
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class 
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,941 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:setup(1534)) - Adding job token for job_123456789_0001 to 
> jobTokenSecretManager
> 2017-08-30 10:12:21,941 WARN  [Thread-49] impl.JobImpl 
> (JobImpl.java:setup(1540)) - Shuffle secret key missing from job credentials. 
> Using job token secret as shuffle secret.
> 2017-08-30 10:12:21,944 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:makeUberDecision(1305)) - Not uberizing job_123456789_0001 
> because: not enabled;
> 2017-08-30 10:12:21,944 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:createMapTasks(1562)) - Input size for job 
> job_123456789_0001 = 0. Number of splits = 2
> 2017-08-30 10:12:21,945 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:createReduceTasks(1579)) - Number of reduces for job 
> job_123456789_0001 = 1
> 2017-08-30 10:12:21,945 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from NEW 
> to INITED
> 2017-08-30 10:12:21,946 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from 
> INITED to SETUP
> 2017-08-30 10:12:21,954 INFO  [CommitterEvent Processor #0] 
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) - 
> Processing the event EventType: JOB_SETUP
> 2017-08-30 10:12:21,978 INFO  [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(1017)) - job_123456789_0001Job Transitioned from 
> SETUP to RUNNING
> 2017-08-30 10:12:21,983 INFO  [Thread-49] event.AsyncDispatcher 
> (AsyncDispatcher.java:register(209)) - Registering class 
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$5
> 2017-08-30 10:12:22,000 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:transition(1953)) - Num completed Tasks: 1
> 2017-08-30 10:12:22,029 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:transition(1953)) - Num completed Tasks: 2
> 2017-08-30 10:12:22,032 INFO  [Thread-49] impl.JobImpl 
> (JobImpl.java:actOnUnusableNode(1354)) - TaskAttempt killed because it ran on 
> unusable node Mock for NodeId, hashCode: 1280187896. 
> AttemptId:attempt_123456789_0001_m_00_0
> 2017-08-30 

[jira] [Created] (MAPREDUCE-7118) Distributed cache conflicts breaks backwards compatability

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-7118:
-

 Summary: Distributed cache conflicts breaks backwards compatability
 Key: MAPREDUCE-7118
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7118
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.1.0, 3.0.0, 3.2.0
Reporter: Jason Lowe
Assignee: Jason Lowe


MAPREDUCE-4503 made distributed cache conflicts break job submission, but this 
was quickly downgraded to a warning in MAPREDUCE-4549.  Unfortunately the 
latter did not go into trunk, so the fix is only in 0.23 and 2.x.  When Oozie, 
Pig, and other downstream projects that can occasionally generate distributed 
cache conflicts move to Hadoop 3.x the workflows that used to work on 0.23 and 
2.x no longer function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7080) Default speculator won't sepculate the last several submitted reduced task if the total task num is large

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-7080.
---
Resolution: Duplicate

Closing as a duplicate of MAPREDUCE-7081.

> Default speculator won't sepculate the last several submitted reduced task if 
> the total task num is large
> -
>
> Key: MAPREDUCE-7080
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7080
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 2.7.5
>Reporter: Zhizhen Hou
>Priority: Major
>
> DefaultSpeculator speculates a task one time. 
> By default, the number of speculators is max(max(10, 0.01 * tasks.size), 0.1 
> * running tasks)
> I  set mapreduce.job.reduce.slowstart.completedmaps = 1 to start reduce after 
> all the map tasks are finished.
> The cluster has 1000 vcores, and the Job has 5000 reduce jobs.
> At first, 1000 reduces tasks can run simultaneously, number of speculators 
> can speculator at most is 0.1 * 1000 = 100 tasks. Reduce tasks with less data 
> can over shortly, and speculator will speculator a task per second by 
> default. The task be speculated execution may be because the more data to be 
> processed. It will speculator  100 tasks within 100 seconds.
> When 4900 reduces is over, If a reduce is executed with a lot of  data be 
> processed and is put on a slow machine. The speculate opportunity is running 
> out, it will not be speculated. It can increase the execution time of job 
> significantly.
> In short, it may waste the speculate opportunity at first only because the 
> execution time of  reduce with less data to be processed as average time. At  
> end of job, there is no speculate opportunity available, especially last 
> several running tasks, judged the number of the running tasks .
>  
> In my opinion, the number of tasks be speculated can be judged by square of 
> finished task percent. Take an example, if ninety percent of  the task is 
> finished, only 0.9*0.9 = 0.81 speculate opportunity can be used. It will 
> leave enough opportunity for latter tasks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7079) TestMRIntermediateDataEncryption is failing in precommit builds

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-7079:
-

 Summary: TestMRIntermediateDataEncryption is failing in precommit 
builds
 Key: MAPREDUCE-7079
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jason Lowe


TestMRIntermediateDataEncryption is either timing out or tearing down the JVM 
which causes the unit tests in jobclient to not pass cleanly during precommit 
builds. From sample precommit console output, note the lack of a test results 
line when the test is run:
{noformat}
[INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s 
- in org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
[INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s 
- in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[...]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:14 h
[INFO] Finished at: 2018-04-12T04:27:06+00:00
[INFO] Final Memory: 24M/594M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[WARNING] The requested profile "native" could not be activated because it does 
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it 
does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There was a timeout or other error 
in the fork -> [Help 1]
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7078) TestPipeApplication is failing in precommit builds

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-7078:
-

 Summary: TestPipeApplication is failing in precommit builds
 Key: MAPREDUCE-7078
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7078
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jason Lowe


TestPipeApplication is either timing out or tearing down the JVM which causes 
the unit tests in jobclient to not pass cleanly during precommit builds.  From 
sample precommit console output, note the lack of a test results line when the 
test is run:
{noformat}
[INFO] Running org.apache.hadoop.mapred.TestIFile
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1 s - in 
org.apache.hadoop.mapred.TestIFile
[INFO] Running org.apache.hadoop.mapred.pipes.TestPipeApplication
[INFO] Running org.apache.hadoop.mapred.pipes.TestPipesNonJavaInputFormat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.02 s - 
in org.apache.hadoop.mapred.pipes.TestPipesNonJavaInputFormat
[...]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:14 h
[INFO] Finished at: 2018-04-12T04:27:06+00:00
[INFO] Final Memory: 24M/594M
[INFO] 
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[WARNING] The requested profile "native" could not be activated because it does 
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it 
does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on 
project hadoop-mapreduce-client-jobclient: There was a timeout or other error 
in the fork -> [Help 1]
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7053) Timed out tasks can fail to produce thread dump

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-7053:
-

 Summary: Timed out tasks can fail to produce thread dump
 Key: MAPREDUCE-7053
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7053
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
Reporter: Jason Lowe


TestMRJobs#testThreadDumpOnTaskTimeout has been failing sporadically recently.  
When the AM times out a task it immediately removes it from the list of known 
tasks and then connects to the NM to request a thread dump followed by a kill.  
If the task heartbeats in after the task has been removed from the list of 
known tasks but before the thread dump signal arrives then the task can exit 
with a "org.apache.hadoop.mapred.Task: Parent died." message and no thread dump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7049) Testcase TestMRJobs#testJobClassloaderWithCustomClasses fails

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-7049.
---
Resolution: Duplicate

> Testcase TestMRJobs#testJobClassloaderWithCustomClasses fails 
> --
>
> Key: MAPREDUCE-7049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7049
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> The testcase TestMRJobs#testJobClassloaderWithCustomClasses fails 
> consistently with this error:
> {noformat}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestMRJobs
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 54.325 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobs
> [ERROR] 
> testJobClassloaderWithCustomClasses(org.apache.hadoop.mapreduce.v2.TestMRJobs)
>   Time elapsed: 10.531 s  <<< FAILURE!
> java.lang.AssertionError: 
> Job status: Application application_1517928628935_0001 failed 2 times due to 
> AM Container for appattempt_1517928628935_0001_02 exited with  exitCode: 1
> Failing this attempt.Diagnostics: [2018-02-06 15:50:38.688]Exception from 
> container-launch.
> Container id: container_1517928628935_0001_02_01
> Exit code: 1
> [2018-02-06 15:50:38.693]Container exited with a non-zero exit code 1. Error 
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [2018-02-06 15:50:38.694]Container exited with a non-zero exit code 1. Error 
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> For more detailed output, check the application tracking page: 
> http://ubuntu:46235/cluster/app/application_1517928628935_0001 Then click on 
> links to logs of each attempt.
> . Failing the application.
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloader(TestMRJobs.java:529)
>   at 
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloaderWithCustomClasses(TestMRJobs.java:477)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Today I found the offending commit with {{git bisect}} and this failure is 
> caused by {{YARN-2185}}.
> The application master fails because of the following error:
> {noformat}
> 2018-02-05 17:15:18,530 DEBUG [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:265)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1694)
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:554)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
>   

[jira] [Created] (MAPREDUCE-7033) Map outputs implicitly rely on permissive umask for shuffle

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-7033:
-

 Summary: Map outputs implicitly rely on permissive umask for 
shuffle
 Key: MAPREDUCE-7033
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7033
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Jason Lowe


Map tasks do not explicitly set the permissions of their output files for 
shuffle.  In a secure cluster the shuffle service is running as a different 
user than the map task, so the output files require group readability in order 
to serve up the data during the shuffle phase.  If the user's UNIX umask is too 
restrictive (e.g.: 077) then the map task's file.out and file.out.index 
permissions can be too restrictive to allow the shuffle handler to access them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7019) java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-7019.
---
Resolution: Invalid

Closing this since I believe the error is coming from the program being 
launched by the streaming job rather than an issue with the streaming framework 
code.  If this is incorrect, please provide details showing where the streaming 
framework code is going awry.


> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed 
> with code 2
> -
>
> Key: MAPREDUCE-7019
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7019
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: shrutika sarda
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6969) TestHSWebApp is failing

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6969:
-

 Summary: TestHSWebApp is failing
 Key: MAPREDUCE-6969
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6969
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Jason Lowe


TestHSWebApp has been failing recently:
{noformat}
Running org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp
Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.57 sec <<< 
FAILURE! - in org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp
testLogsViewBadStartEnd(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp)  
Time elapsed: 0.076 sec  <<< FAILURE!
org.mockito.exceptions.verification.junit.ArgumentsAreDifferent: 
Argument(s) are different! Wanted:
printWriter.write(
"Invalid log end value: bar"
);
-> at 
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewBadStartEnd(TestHSWebApp.java:261)
Actual invocation has different arguments:
printWriter.write(
"http://www.w3.org/TR/html4/strict.dtd;>"
);
-> at 
org.apache.hadoop.yarn.webapp.view.TextView.echoWithoutEscapeHtml(TextView.java:62)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewBadStartEnd(TestHSWebApp.java:261)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6968) Staging directory erasure coding config property has a typo

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6968:
-

 Summary: Staging directory erasure coding config property has a 
typo
 Key: MAPREDUCE-6968
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6968
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 3.0.0-beta1
Reporter: Jason Lowe
Assignee: Jason Lowe


TestMapreduceConfigFields has been failing since MAPREDUCE-6954. 
MRJobConfig#MR_AM_STAGING_DIR_ERASURECODING_ENABLED is defined as 
"yarn.app.mapreduce.am.staging-direrasurecoding.enabled"  but the property is 
listed as "yarn.app.mapreduce.am.staging-dir.erasurecoding.enabled" in 
mapred-default.xml.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6959) Understanding on process to start contribution

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6959.
---
Resolution: Invalid

JIRA is for tracking features and bugs in Hadoop and not for general support.  
Questions such as these can be directed to the [mailing 
lists|http://hadoop.apache.org/mailing_lists.html].  Specifically if you're 
interested in contributing I highly recommend checking out the [How To 
Contribute|https://wiki.apache.org/hadoop/HowToContribute] wiki page.

Note that https://github.com/apache/hadoop-mapreduce is a mirror of just the 
MapReduce code from what looks like Hadoop 1.x or even earlier code that is no 
longer supported.  All active development is on Hadoop 2. x and Hadoop 3.x.


> Understanding on process to start contribution
> --
>
> Key: MAPREDUCE-6959
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6959
> Project: Hadoop Map/Reduce
>  Issue Type: Wish
>Reporter: Mehul
>Priority: Trivial
>
> I was trying to find process/steps to start with contribution into following 
> repo i.e. https://github.com/apache/hadoop-mapreduce. Can someone please help 
> with the detail so that I can create appropriate git/jira issue and start 
> woking on it?
> Any direction would be really appreciated!
> Thanks,
> Mehul



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6958) Shuffle audit logger should log size of shuffle transfer

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6958:
-

 Summary: Shuffle audit logger should log size of shuffle transfer
 Key: MAPREDUCE-6958
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Minor


The shuffle audit logger currently logs the job ID and reducer ID but nothing 
about the size of the requested transfer.  It calculates this as part of the 
HTTP response headers, so it would be trivial to log the response size.  This 
would be very valuable for debugging network traffic storms from the shuffle 
handler.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6952) Using DistributedCache.addFileToClasspath with a rename fragment fails during job submit

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6952:
-

 Summary: Using DistributedCache.addFileToClasspath with a rename 
fragment fails during job submit
 Key: MAPREDUCE-6952
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6952
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.8.1, 2.7.4
Reporter: Jason Lowe


Calling DistributedCache.addFileToClasspath with a Path that specifies a URI 
fragment, used to rename the file during localization, causes job submission to 
fail with a FileNotFoundException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-6641) TestTaskAttempt fails in trunk

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-6641:
---

Seeing this fail the same way in 2.8 builds as well.  Unfortunately since the 
fix uses lambdas I can't just cherry-pick the fix down to other branches.  
Reopening so Jenkins can comment on a branch-2 version of the patch.

> TestTaskAttempt fails in trunk
> --
>
> Key: MAPREDUCE-6641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6641
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Tsuyoshi Ozawa
>Assignee: Haibo Chen
> Fix For: 3.0.0-alpha1
>
> Attachments: mapreduce6641.001.patch, mapreduce6641.002.patch, 
> MAPREDUCE-6641-branch-2.002.patch, 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt-output.txt
>
>
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.917 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> testMRAppHistoryForTAFailedInAssigned(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>   Time elapsed: 12.732 sec  <<< FAILURE!
> java.lang.AssertionError: No Ta Started JH Event
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testTaskAttemptAssignedKilledHistory(TestTaskAttempt.java:388)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned(TestTaskAttempt.java:177)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6933) Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6933.
---
Resolution: Duplicate

> Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED
> ---
>
> Key: MAPREDUCE-6933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6933
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.1, 3.0.0-alpha4
>Reporter: lujie
>
> When I run a job on 0.23.1, I found a InvalidStateTransitonException:
> {code:java}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> TA_CONTAINER_LAUNCH_FAILED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:926)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:870)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:862)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> After I manually analyse the code of 3.0.0,I think this error may still 
> exists.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6916) History server scheduling tasks at fixed rate can be problematic when those tasks are slow

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6916:
-

 Summary: History server scheduling tasks at fixed rate can be 
problematic when those tasks are slow
 Key: MAPREDUCE-6916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.7.4
Reporter: Jason Lowe


The job history server currently schedules both the task of moving jobs from 
intermediate to done and the task of cleaning jobs at a fixed rate.  If those 
tasks take longer than the rate period to execute then a backlog of 
to-be-scheduled tasks can build up and cause a long storm of them to execute 
later when the blockage clears.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6909) LocalJobRunner fails when run on a node from multiple users

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6909:
-

 Summary: LocalJobRunner fails when run on a node from multiple 
users
 Key: MAPREDUCE-6909
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6909
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.8.1
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker


MAPREDUCE-5762 removed mapreduce.jobtracker.staging.root.dir from 
mapred-default.xml but the property is still being used by LocalJobRunner and 
the code default value does *not* match the value that was removed from 
mapred-default.xml.  This broke the use case where multiple users are running 
local mode jobs on the same node, since they now default to the same directory 
in /tmp.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6898) TestKill.testKillTask is flaky

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6898.
---
   Resolution: Duplicate
Fix Version/s: (was: 2.8.2)
   (was: 3.0.0-alpha4)
   (was: 2.9.0)

> TestKill.testKillTask is flaky
> --
>
> Key: MAPREDUCE-6898
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6898
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6898-001.patch
>
>
> TestKill.testKillTask() can fail if the async dispatcher thread is slower 
> than the test's thread.
> {noformat}
> 2017-05-26 11:43:26,532 INFO  [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(1006)) - job_0_Job Transitioned from INITED to SETUP
> Job State is : RUNNING
> Job State is : RUNNING Waiting for state : SUCCEEDED   map progress : 0.0   
> reduce progress : 0.0
> 2017-05-26 11:43:26,538 INFO  [CommitterEvent Processor #0] 
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) - 
> Processing the event EventType: JOB_SETUP
> 2017-05-26 11:43:26,540 INFO  [AsyncDispatcher event handler] impl.TaskImpl 
> (TaskImpl.java:handle(661)) - task_0__m_00 Task Transitioned from NEW 
> to KILLED
> 2017-05-26 11:43:26,540 ERROR [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(998)) - Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> JOB_TASK_COMPLETED at SETUP
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1366)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1362)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-05-26 11:43:26,541 INFO  [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(1006)) - job_0_Job Transitioned from SETUP to ERROR
> 2017-05-26 11:43:26,542 INFO  [AsyncDispatcher event handler] app.MRAppMaster 
> (MRAppMaster.java:serviceStop(978)) - Skipping cleaning up the staging dir. 
> assuming AM will be retried.
> {noformat}
> We have to wait until the job's internal state is 
> {{JobInternalState.RUNNING}} and not {{JobInternalState.SETUP}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-6898) TestKill.testKillTask is flaky

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-6898:
---

No worries, I'll revert and mark this as a duplicate of MAPREDUCE-6815.

> TestKill.testKillTask is flaky
> --
>
> Key: MAPREDUCE-6898
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6898
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: MAPREDUCE-6898-001.patch
>
>
> TestKill.testKillTask() can fail if the async dispatcher thread is slower 
> than the test's thread.
> {noformat}
> 2017-05-26 11:43:26,532 INFO  [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(1006)) - job_0_Job Transitioned from INITED to SETUP
> Job State is : RUNNING
> Job State is : RUNNING Waiting for state : SUCCEEDED   map progress : 0.0   
> reduce progress : 0.0
> 2017-05-26 11:43:26,538 INFO  [CommitterEvent Processor #0] 
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) - 
> Processing the event EventType: JOB_SETUP
> 2017-05-26 11:43:26,540 INFO  [AsyncDispatcher event handler] impl.TaskImpl 
> (TaskImpl.java:handle(661)) - task_0__m_00 Task Transitioned from NEW 
> to KILLED
> 2017-05-26 11:43:26,540 ERROR [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(998)) - Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> JOB_TASK_COMPLETED at SETUP
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
>   at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1366)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1362)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
>   at java.lang.Thread.run(Thread.java:745)
> 2017-05-26 11:43:26,541 INFO  [AsyncDispatcher event handler] impl.JobImpl 
> (JobImpl.java:handle(1006)) - job_0_Job Transitioned from SETUP to ERROR
> 2017-05-26 11:43:26,542 INFO  [AsyncDispatcher event handler] app.MRAppMaster 
> (MRAppMaster.java:serviceStop(978)) - Skipping cleaning up the staging dir. 
> assuming AM will be retried.
> {noformat}
> We have to wait until the job's internal state is 
> {{JobInternalState.RUNNING}} and not {{JobInternalState.SETUP}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6869) org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6869.
---
Resolution: Not A Bug

Closing this since it does not appear to be a problem in Hadoop.  Please reopen 
with additional evidence if you find otherwise.

> org.apache.hadoop.mapred.ShuffleHandler: Shuffle error in populating headers :
> --
>
> Key: MAPREDUCE-6869
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6869
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, yarn
>Affects Versions: 2.6.0
> Environment: hadoop 2.6.0-cdh5.8.2
>Reporter: 翟玉勇
>Priority: Minor
>
> nodemanager log
> 2017-03-25 21:07:03,071 ERROR org.apache.hadoop.mapred.ShuffleHandler: 
> Shuffle error in populating headers :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
> usercache/master/appcache/application_1489067586592_930490/output/attempt_1489067586592_930490_m_002811_0/file.out.index
>  in any of the configured local directories
> at 
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:488)
> at 
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:165)
> at 
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:1000)
> at 
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1022)
> at 
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:908)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at 
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at 
> org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at 
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
> at 
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
> at 
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> at 
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at 
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at 
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at 
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at 
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> reduce log
> 2017-03-25 05:40:33,919 WARN [fetcher#2] 
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id 
> 

[jira] [Created] (MAPREDUCE-6763) Shuffle server listen queue is too small

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6763:
-

 Summary: Shuffle server listen queue is too small
 Key: MAPREDUCE-6763
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6763
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Jason Lowe
Assignee: Jason Lowe


ShuffleHandler doesn't specify a listen queue length for the server port, so it 
ends up getting the default listen queue length of 50.  This is too small to 
handle bursts of shuffle traffic on large clusters.  It's also inconsistent 
with the default Hadoop uses for RPC servers (default=128).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-3294) Log the reason for killing a task during speculative execution

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-3294.
---
Resolution: Duplicate

This was fixed by MAPREDUCE-5692.

> Log the reason for killing a task during speculative execution
> --
>
> Key: MAPREDUCE-3294
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3294
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Ramya Sunil
>
> The reason for killing a speculated task has to be logged. Currently, a 
> speculated task is killed with a note of "Container killed by the 
> ApplicationMaster. Container killed on request. Exit code is 137" which is 
> not very useful. Better logging of this message stating the task was killed 
> due to completion of its speculative task would be useful.
> Also, this message is lost once the app is moved to history. All we are left 
> with is a list of killed tasks without a reason being notified to the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-4758) jobhistory web ui not showing correct # failed reducers

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-4758.
---
Resolution: Duplicate

This is a duplicate of MAPREDUCE-5982 which was fixed in 2.7.2 and 2.6.4.

> jobhistory web ui not showing correct # failed reducers
> ---
>
> Key: MAPREDUCE-4758
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4758
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, webapps
>Affects Versions: 0.23.4
>Reporter: Thomas Graves
>
> we had a job fail due to a reducer failing 4 times.  Unfortunately the job 
> history UI didn't show  this particular failed reducer which lead to 
> confusion as to why the job failed. 
> This reducer failed to launch all 4 task attempts with a Token Expiration 
> error and the jobhistory file only gets an event when the task attempt 
> transitions to launched.  The webapp JobInfo object only counts the task 
> attempts in the jobhistory file to display under the "Attempt Type" table, so 
> since this task didn't have an attempt with it, it did show it on the UI.
> We need to reconcile the task list with the task attempts or also shows more 
> stats for the tasks vs task attempts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6625:
-

 Summary: TestCLI#testGetJob fails occasionally
 Key: MAPREDUCE-6625
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Jason Lowe


Lately TestCLI has been failing sometimes in precommit builds:
{noformat}
Running org.apache.hadoop.mapreduce.tools.TestCLI
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<< 
FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI)  Time elapsed: 0.037 sec  
<<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6623) TestRMNMInfo and TestNetworkedJob fails in trunk

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6623.
---
Resolution: Duplicate

Resolving as a duplicate per the previous comment.

> TestRMNMInfo and TestNetworkedJob fails in trunk
> 
>
> Key: MAPREDUCE-6623
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6623
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Eric Badger
>
> TestRMNMInfo:
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestRMNMInfo
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.347 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestRMNMInfo
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo)  Time elapsed: 
> 1.572 sec  <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but 
> was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at 
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {code}
> TestNetworkedJob
> {code}
> testNetworkedJob:174 expected:<[[Thu Jan 28 22:41:20 + 2016] Application 
> is Activated, waiting for resources to be assigned for AM.  Details : AM 
> Partition =  ; Partition Resource =  vCores:16> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used 
> capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> but was:<[]>
>   TestRMNMInfo.testRMNMInfo:111 Unexpected number of live nodes: expected:<4> 
> but was:<0>
> {code}
> JDK version: JDK v1.8.0_66



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6599) ResourceManager crash due to scheduling opportunity overflow

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6599:
-

 Summary: ResourceManager crash due to scheduling opportunity 
overflow
 Key: MAPREDUCE-6599
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6599
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.1
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Critical


If a resource request lingers long enough unsatisfied then the scheduling 
opportunities count for the request can overflow and cause an RM crash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-4938) Job submission to unknown queue can leave staging directory behind

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-4938.
---
Resolution: Duplicate

> Job submission to unknown queue can leave staging directory behind
> --
>
> Key: MAPREDUCE-4938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4938
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client
>Affects Versions: 2.0.3-alpha, 0.23.5
>Reporter: Jason Lowe
>
> There is a race where submitting a job to an unknown queue can appear to 
> succeed to the client and then subsequently fail later.  Since there was no 
> AM ever launched, there was nothing left to cleanup the staging directory.  
> At that point the client is the only thing that can cleanup the staging 
> directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6472:
-

 Summary: MapReduce AM should have java.io.tmpdir=./tmp to be 
consistent with tasks
 Key: MAPREDUCE-6472
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Jason Lowe


MapReduceChildJVM.getVMCommand ensures that all tasks have 
-Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
setting.  It should also use the same tmpdir setting to avoid cases where the 
AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6413) TestLocalJobSubmission is failing

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6413:
-

 Summary: TestLocalJobSubmission is failing
 Key: MAPREDUCE-6413
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6413
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 2.7.1
Reporter: Jason Lowe


ThestLocalJobSubmission.testLocalJobLibjarsOption is failing with 
java.net.UnknownHostException: testcluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6355) 2.5 client cannot communicate with 2.5 job on 2.6 cluster

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6355:
-

 Summary: 2.5 client cannot communicate with 2.5 job on 2.6 cluster
 Key: MAPREDUCE-6355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6355
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Jason Lowe


Trying to run a job on a Hadoop 2.6 cluster from a Hadoop 2.5 client submitting 
a job that uses Hadoop 2.5 jars results in a job that succeeds but the client 
cannot communicate with the AM while the job is running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6324) Uber jobs fail to update AMRM token when it rolls over

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6324:
-

 Summary: Uber jobs fail to update AMRM token when it rolls over
 Key: MAPREDUCE-6324
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6324
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.6.0
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker


When the RM rolls a new AMRM master key the AMs are supposed to receive a new 
AMRM token on subsequent heartbeats between the time when the new key is rolled 
and when it is activated.  This is not occurring for uber jobs.  If the 
connection to the RM needs to be re-established after the new key is activated 
(e.g.: RM restart or network hiccup) then the uber job AM will be unable to 
reconnect to the RM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6303) Read timeout when retrying a fetch error can be fatal to a reducer

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6303:
-

 Summary: Read timeout when retrying a fetch error can be fatal to 
a reducer
 Key: MAPREDUCE-6303
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6303
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Jason Lowe
Priority: Blocker


If a reducer encounters an error trying to fetch from a node then encounters a 
read timeout when trying to re-establish the connection then the reducer can 
fail.  The read timeout exception can leak to the top of the Fetcher thread 
which will cause the reduce task to teardown.  This type of error can repeat 
across reducer attempts causing jobs to fail due to a single bad node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6279:
-

 Summary: AM should explicity exit JVM after all services have 
stopped
 Key: MAPREDUCE-6279
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Jason Lowe


Occasionally the MapReduce AM can get stuck trying to shut down.  
MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, 
but this can also occur with uber jobs if the task code inadvertently leaves 
non-daemon threads lingering.

We should explicitly shutdown the JVM after the MapReduce AM has unregistered 
and all services have been stopped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6263) Large jobs can lose history when killed due to brief client timeout

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6263:
-

 Summary: Large jobs can lose history when killed due to brief 
client timeout
 Key: MAPREDUCE-6263
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6263
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.6.0
Reporter: Jason Lowe


YARNRunner connects to the AM to send the kill job command then waits a 
hardcoded 10 seconds for the job to enter a terminal state.  If the job fails 
to enter a terminal state in that time then YARNRunner will tell YARN to kill 
the application forcefully.  The latter type of kill usually results in no job 
history, since the AM process is killed forcefully.

Ten seconds can be too short for large jobs in a large cluster, as it takes 
time to connect to all the nodemanagers, process the state machine events, and 
copy a large jhist file.  The timeout should be more lenient or configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6261) NullPointerException if MapOutputBuffer.flush invoked twice

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6261:
-

 Summary: NullPointerException if MapOutputBuffer.flush invoked 
twice
 Key: MAPREDUCE-6261
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6261
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.5.0
Reporter: Jason Lowe


MapOutputBuffer.flush will throw an NPE if it is invoked twice, since it 
blindly assumes kvbuffer is not null yet sets kvbuffer to null towards the end 
of the method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-5727) History server web page can filter without showing filter keyword

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5727.
---
Resolution: Duplicate

This is the same issue as described in YARN-2238, and there's more discussion 
there.

 History server web page can filter without showing filter keyword
 -

 Key: MAPREDUCE-5727
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5727
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, webapps
Affects Versions: 2.3.0
Reporter: Jason Lowe

 I loaded up a job conf page on the history server and used one of the search 
 boxes to narrow the results.  I then navigated to other pages (e.g.: map 
 tasks, logs, etc.) then navigated back to the job conf page using the job 
 configuration link on the left side of the page.  When I arrived it promptly 
 showed me just a few conf entries (the ones I had searched for earlier) but 
 my search term was missing.  At first glance it looked like those were the 
 only entries in the entire job conf, which can be very confusing.  Somehow 
 the search term is being remembered but not replotted when the configuration 
 page is revisited.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6230) MR AM does not survive RM restart if RM activated a new AMRM secret key

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6230:
-

 Summary: MR AM does not survive RM restart if RM activated a new 
AMRM secret key
 Key: MAPREDUCE-6230
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6230
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker


A MapReduce AM will fail to reconnect to an RM that performed restart in the 
following scenario:

# MapReduce job launched with AMRM token generated from AMRM secret X
# RM rolls new AMRM secret Y and activates the new key
# RM performs a work-preserving restart
# MapReduce job AM now unable to connect to RM with Invalid AMRMToken 
exception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6225) Fix new findbug warnings in hadoop-mapreduce-client-core

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6225:
-

 Summary: Fix new findbug warnings in hadoop-mapreduce-client-core
 Key: MAPREDUCE-6225
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6225
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Jason Lowe


Recent precommit builds in hadoop-mapreduce-client-core are flagging findbug 
warnings that appear to be new with the recent findbugs upgrade.  These need to 
be cleaned up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6219) Reduce memory required for FileInputFormat located status optimization

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6219:
-

 Summary: Reduce memory required for FileInputFormat located status 
optimization
 Key: MAPREDUCE-6219
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6219
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Jason Lowe
Priority: Minor


MAPREDUCE-1981 introduced an optimization to drastically reduce the number of 
namenode operations required to compute input splits when processing a 
directory.  However it requires more memory to perform this optimization as it 
retains the full LocatedFileStatus object for all input files while computing 
the splits.  This can lead to odd situations for users where using a directory 
as input can run the job client out of heap space but using directory/* as the 
input spec allows it to run within the original heap space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6172) TestDbClasses timeouts are too aggressive

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6172:
-

 Summary: TestDbClasses timeouts are too aggressive
 Key: MAPREDUCE-6172
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6172
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 2.6.0
Reporter: Jason Lowe
Priority: Minor


Some of the TestDbClasses test timeouts are only 1 second, and some of those 
tests perform disk I/O which could easily exceed the test timeout if the disk 
is busy or there's some other hiccup on the system at the time.  We should 
increase these timeouts to something more reasonable (i.e.: 10 or 20 seconds).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6161) mapred hsadmin command missing from trunk

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6161:
-

 Summary: mapred hsadmin command missing from trunk
 Key: MAPREDUCE-6161
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6161
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: scripts
Affects Versions: trunk
Reporter: Jason Lowe


The hsadmin subcommand of the mapred script is no longer present in trunk. It 
is present in branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6159) No log of JobHistory found in all logs files

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6159.
---
Resolution: Invalid

The JobHistoryEventHandler is code that runs in the ApplicationMaster rather 
than the job history server.  You'll find those log messages in the AM logs of 
individual jobs which are either aggregated to HDFS (by default) or left on the 
nodes the AMs ran on if log aggregation is disabled.

 No log of JobHistory found in all logs files
 

 Key: MAPREDUCE-6159
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6159
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, mrv2
Affects Versions: 2.2.0
 Environment: Hadoop-2.2.0
Reporter: JasonZhu

 I intend to dig into 'mapreduce.jobhistory.intermediate-done-dir' argument, 
 the position of which is at `JHAdminConfig:73`, to get some comprehension on 
 history server. This argument is referenced at 
 `JobHistoryEventHandler.moveToDoneNow()`, where history server moves job 
 summary file 
 from $[yarn.app.mapreduce.am.staging-dir]/$[user]/.staging to 
 $[mapreduce.jobhistory.intermediate-done-dir]/$[user]. 
 The following code snippet in `moveToDoneNow()` will definitely write some 
 logs out to log file, but I can found no any sign of it in all logs in 
 $HADOOP_LOG_DIR via command `grep Copied to done location *`.
 if (copied)
 LOG.info(Copied to done location:  + toPath);
 else 
 LOG.info(copy failed);
 Is there anything that I missed?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6141) History server leveldb recovery store

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6141:
-

 Summary: History server leveldb recovery store
 Key: MAPREDUCE-6141
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Reporter: Jason Lowe
Assignee: Jason Lowe


It would be nice to have a leveldb option to the job history server recovery 
store.  Leveldb would provide some benefits over the existing filesystem store 
such as better support for atomic operations, fewer I/O ops per state update, 
and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6119) Ability to disable node update processing in MR AM

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6119:
-

 Summary: Ability to disable node update processing in MR AM
 Key: MAPREDUCE-6119
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6119
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mr-am
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Jason Lowe






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (MAPREDUCE-6114) TestMRCJCFileInputFormat#testAddInputPath fails in trunk

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6114.
---
Resolution: Duplicate

Dup of MAPREDUCE-6094.

 TestMRCJCFileInputFormat#testAddInputPath fails in trunk
 

 Key: MAPREDUCE-6114
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6114
 Project: Hadoop Map/Reduce
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 This can be reproduced locally:
 {code}
 Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.474 sec  
 FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
 testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
   Time elapsed: 0.86 sec   ERROR!
 java.io.IOException: No FileSystem for scheme: s3
   at 
 org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
   at 
 org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
   at 
 org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
   at 
 org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6075) HistoryServerFileSystemStateStore can create zero-length files

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6075:
-

 Summary: HistoryServerFileSystemStateStore can create zero-length 
files
 Key: MAPREDUCE-6075
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6075
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Jason Lowe
Assignee: Jason Lowe


When the history server state store writes a token file it uses 
IOUtils.cleanup() to close the file which will silently ignore errors.  This 
can lead to empty token files in the state store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6021) MR AM should add working directory to LD_LIBRARY_PATH

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6021:
-

 Summary: MR AM should add working directory to LD_LIBRARY_PATH
 Key: MAPREDUCE-6021
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6021
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.4.1
Reporter: Jason Lowe


Tasks implicitly pick up shared libraries added to the job because the task 
launch context explicitly adds the container working directory to 
LD_LIBRARY_PATH.  However the same is not done for the AM container which is 
inconsistent.  User code can run in the AM via output committer, speculator, 
uber job, etc., so the AM's LD_LIBRARY_PATH should have the container work 
directory for consistency with tasks.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-6022) map_input_file is missing from streaming job environment

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6022:
-

 Summary: map_input_file is missing from streaming job environment
 Key: MAPREDUCE-6022
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6022
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Jason Lowe


When running a streaming job the 'map_input_file' environment variable is not 
being set.  This property is deprecated, but in the past deprecated properties 
still appeared in a stream job's environment.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-6010) HistoryServerFileSystemStateStore fails to update tokens

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6010:
-

 Summary: HistoryServerFileSystemStateStore fails to update tokens
 Key: MAPREDUCE-6010
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6010
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Jason Lowe


When token recovery is enabled and the file system state store is being used 
then tokens fail to be updated due to a rename destination conflict.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-6011) Improve history server behavior during a recovery error

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6011:
-

 Summary: Improve history server behavior during a recovery error
 Key: MAPREDUCE-6011
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6011
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Jason Lowe


Currently when the history server encounters an error during recovery it is 
fatal without specific details on the error (e.g. which token was involved 
during the recovery error).  We should either allow the history server to 
proceed past recovery errors or provide more specifics on the offending token 
involved in the fatal error to aid in manual recovery.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5928) Deadlock allocating containers for mappers and reducers

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5928.
---

Resolution: Duplicate

 Deadlock allocating containers for mappers and reducers
 ---

 Key: MAPREDUCE-5928
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5928
 Project: Hadoop Map/Reduce
  Issue Type: Bug
 Environment: Hadoop 2.4.0 (as packaged by HortonWorks in HDP 2.1.2)
Reporter: Niels Basjes
 Attachments: AM-MR-syslog - Cleaned.txt.gz, Cluster fully 
 loaded.png.jpg, MR job stuck in deadlock.png.jpg


 I have a small cluster consisting of 8 desktop class systems (1 master + 7 
 workers).
 Due to the small memory of these systems I configured yarn as follows:
 {quote}
 yarn.nodemanager.resource.memory-mb = 2200
 yarn.scheduler.minimum-allocation-mb = 250
 {quote}
 On my client I did
 {quote}
 mapreduce.map.memory.mb = 512
 mapreduce.reduce.memory.mb = 512
 {quote}
 Now I run a job with 27 mappers and 32 reducers.
 After a while I saw this deadlock occur:
 - All nodes had been filled to their maximum capacity with reducers.
 - 1 Mapper was waiting for a container slot to start in.
 I tried killing reducer attempts but that didn't help (new reducer attempts 
 simply took the existing container).
 *Workaround*:
 I set this value from my job. The default value is 0.05 (= 5%)
 {quote}
 mapreduce.job.reduce.slowstart.completedmaps = 0.99f
 {quote}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (MAPREDUCE-5927) Getting following error

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-5927:
---


 Getting following error
 ---

 Key: MAPREDUCE-5927
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Reporter: Kedar Dixit
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker

 Hi,
 I am getting following error, while running application on cluser -
 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option 
 parsing not performed. Implement the Tool interface and execute your 
 application with ToolRunner to remedy this.
 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1
 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. 
 Instead, use mapreduce.job.user.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. 
 Instead, use mapreduce.job.jar
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is 
 deprecated. Instead, use mapreduce.job.reduces
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class 
 is deprecated. Instead, use mapreduce.job.output.value.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is 
 deprecated. Instead, use mapreduce.job.map.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is 
 deprecated. Instead, use mapreduce.job.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class 
 is deprecated. Instead, use mapreduce.job.inputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is 
 deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is 
 deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: 
 mapreduce.outputformat.class is deprecated. Instead, use 
 mapreduce.job.outputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is 
 deprecated. Instead, use mapreduce.job.maps
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is 
 deprecated. Instead, use mapreduce.job.output.key.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is 
 deprecated. Instead, use mapreduce.job.working.dir
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
 job_1402913701967_0006
 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application 
 application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032
 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: 
 http://gs-1695:8088/proxy/application_1402913701967_0006/
 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in 
 uber mode : false
 14/06/16 16:21:54 INFO mapreduce.Job:  map 0% reduce 0%
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with 
 state FAILED due to: Application application_1402913701967_0006 failed 2 
 times due to AM Container for appattempt_1402913701967_0006_02 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at 
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 .Failing this attempt.. Failing the application.
 14/06/16 16:21:54 INFO mapreduce.Job: Counters: 0
 Can you please help me in fixing this ?
 Thanks,
 ~Kedar



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5927) Getting following error

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5927.
---

Resolution: Fixed

 Getting following error
 ---

 Key: MAPREDUCE-5927
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Reporter: Kedar Dixit
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker

 Hi,
 I am getting following error, while running application on cluser -
 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option 
 parsing not performed. Implement the Tool interface and execute your 
 application with ToolRunner to remedy this.
 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1
 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. 
 Instead, use mapreduce.job.user.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. 
 Instead, use mapreduce.job.jar
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is 
 deprecated. Instead, use mapreduce.job.reduces
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class 
 is deprecated. Instead, use mapreduce.job.output.value.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is 
 deprecated. Instead, use mapreduce.job.map.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is 
 deprecated. Instead, use mapreduce.job.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class 
 is deprecated. Instead, use mapreduce.job.inputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is 
 deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is 
 deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: 
 mapreduce.outputformat.class is deprecated. Instead, use 
 mapreduce.job.outputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is 
 deprecated. Instead, use mapreduce.job.maps
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is 
 deprecated. Instead, use mapreduce.job.output.key.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is 
 deprecated. Instead, use mapreduce.job.working.dir
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
 job_1402913701967_0006
 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application 
 application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032
 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: 
 http://gs-1695:8088/proxy/application_1402913701967_0006/
 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in 
 uber mode : false
 14/06/16 16:21:54 INFO mapreduce.Job:  map 0% reduce 0%
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with 
 state FAILED due to: Application application_1402913701967_0006 failed 2 
 times due to AM Container for appattempt_1402913701967_0006_02 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at 
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 .Failing this attempt.. Failing the application.
 14/06/16 16:21:54 INFO mapreduce.Job: Counters: 0
 Can you please help me in fixing this ?
 Thanks,
 ~Kedar



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5927) Getting following error

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5927.
---

Resolution: Invalid

This is a general support question better asked on the u...@hadoop.apache.org 
list.  JIRA is for tracking bugs and features in Hadoop and not a general user 
support channel.

In this case the ApplicationMaster is crashing shortly after startup.  You'll 
need to examine the ApplicationMaster log to determine what happened -- click 
on the tracking URL and then from there go to the AM logs link or you can also 
use the yarn logs command if log aggregation is enabled on your cluster.

 Getting following error
 ---

 Key: MAPREDUCE-5927
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Reporter: Kedar Dixit
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker

 Hi,
 I am getting following error, while running application on cluser -
 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option 
 parsing not performed. Implement the Tool interface and execute your 
 application with ToolRunner to remedy this.
 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1
 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. 
 Instead, use mapreduce.job.user.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. 
 Instead, use mapreduce.job.jar
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is 
 deprecated. Instead, use mapreduce.job.reduces
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class 
 is deprecated. Instead, use mapreduce.job.output.value.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is 
 deprecated. Instead, use mapreduce.job.map.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is 
 deprecated. Instead, use mapreduce.job.name
 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class 
 is deprecated. Instead, use mapreduce.job.inputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is 
 deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is 
 deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
 14/06/16 16:21:49 INFO Configuration.deprecation: 
 mapreduce.outputformat.class is deprecated. Instead, use 
 mapreduce.job.outputformat.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is 
 deprecated. Instead, use mapreduce.job.maps
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is 
 deprecated. Instead, use mapreduce.job.output.key.class
 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is 
 deprecated. Instead, use mapreduce.job.working.dir
 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
 job_1402913701967_0006
 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application 
 application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032
 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: 
 http://gs-1695:8088/proxy/application_1402913701967_0006/
 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in 
 uber mode : false
 14/06/16 16:21:54 INFO mapreduce.Job:  map 0% reduce 0%
 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with 
 state FAILED due to: Application application_1402913701967_0006 failed 2 
 times due to AM Container for appattempt_1402913701967_0006_02 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at 
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

[jira] [Resolved] (MAPREDUCE-5923) org.apache.hadoop.mapred.pipes.TestPipeApplication timeouts intermittently

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5923.
---

Resolution: Duplicate

This is a duplicate of MAPREDUCE-5868.

 org.apache.hadoop.mapred.pipes.TestPipeApplication timeouts intermittently
 --

 Key: MAPREDUCE-5923
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5923
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: trunk
Reporter: Chen He
Priority: Minor





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (MAPREDUCE-5830) HostUtil.getTaskLogUrl is not backwards binary compatible with 2.3

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-5830:
---


Reopening this, as we should address older Hive versions.

 HostUtil.getTaskLogUrl is not backwards binary compatible with 2.3
 --

 Key: MAPREDUCE-5830
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5830
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Jason Lowe
Priority: Blocker

 HostUtil.getTaskLogUrl used to have a signature like this in Hadoop 2.3.0 and 
 earlier:
 public static String getTaskLogUrl(String taskTrackerHostName, String 
 httpPort, String taskAttemptID)
 but now has a signature like this:
 public static String getTaskLogUrl(String scheme, String taskTrackerHostName, 
 String httpPort, String taskAttemptID)
 This breaks source and binary backwards-compatibility.  MapReduce and Hive 
 both have references to this, so their jars compiled against 2.3 or earlier 
 do not work on 2.4.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5891) Improved shuffle error handling across NM restarts

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5891:
-

 Summary: Improved shuffle error handling across NM restarts
 Key: MAPREDUCE-5891
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5891
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Jason Lowe


To minimize the number of map fetch failures reported by reducers across an NM 
restart it would be nice if reducers only reported a fetch failure after trying 
for at specified period of time to retrieve the data.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5888) Failed job leaves hung AM after it unregisters

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5888:
-

 Summary: Failed job leaves hung AM after it unregisters 
 Key: MAPREDUCE-5888
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5888
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Jason Lowe


When a job fails the AM hangs during shutdown.  A non-daemon thread pool 
executor thread prevents the JVM teardown from completing, and the AM lingers 
on the cluster for the AM expiry interval in the FINISHING state until 
eventually the RM expires it and kills the container.  If application limits on 
the queue are relatively low (e.g.: small queue or small cluster) this can 
cause unnecessary delays in resource scheduling on the cluster.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5885) build/test/test.mapred.spill causes release audit warnings

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5885:
-

 Summary: build/test/test.mapred.spill causes release audit warnings
 Key: MAPREDUCE-5885
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5885
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: trunk
Reporter: Jason Lowe


Multiple unit tests are creating files under 
hadoop-mapreduce-client-jobclient/build/test/test.mapred.spill which are 
causing release audit warnings during Jenkins patch precommit builds.  In 
addition to being in a poor location for test output and not cleaning up after 
the test, there are multiple tests using this location which will cause 
conflicts if tests are run in parallel.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5868) TestPipeApplication causing nightly build to fail

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5868:
-

 Summary: TestPipeApplication causing nightly build to fail
 Key: MAPREDUCE-5868
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5868
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: trunk
Reporter: Jason Lowe


TestPipeApplication appears to be timing out which causes the nightly build to 
fail.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-4852) Reducer should not signal fetch failures for disk errors on the reducer's side

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-4852.
---

Resolution: Duplicate

This was fixed by MAPREDUCE-5251.

 Reducer should not signal fetch failures for disk errors on the reducer's side
 --

 Key: MAPREDUCE-4852
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4852
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Jason Lowe

 Ran across a case where a reducer ran on a node where the disks were full, 
 leading to an exception like this during the shuffle fetch:
 {noformat}
 2012-12-05 09:07:28,749 INFO [fetcher#25] 
 org.apache.hadoop.mapreduce.task.reduce.MergeManager: 
 attempt_1352354913026_138167_m_000654_0: Shuffling to disk since 235056188 is 
 greater than maxSingleShuffleLimit (155104064)
 2012-12-05 09:07:28,755 INFO [fetcher#25] 
 org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#25 failed to read 
 map headerattempt_1352354913026_138167_m_000654_0 decomp: 235056188, 101587629
 org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any 
 valid local directory for 
 output/attempt_1352354913026_138167_r_000189_0/map_654.out
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:398)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.mapred.YarnOutputFiles.getInputFileForWrite(YarnOutputFiles.java:213)
   at 
 org.apache.hadoop.mapreduce.task.reduce.MapOutput.init(MapOutput.java:81)
   at 
 org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:245)
   at 
 org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:348)
   at 
 org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:283)
   at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:155)
 2012-12-05 09:07:28,755 WARN [fetcher#25] 
 org.apache.hadoop.mapreduce.task.reduce.Fetcher: copyMapOutput failed for 
 tasks [attempt_1352354913026_138167_m_000654_0]
 2012-12-05 09:07:28,756 INFO [fetcher#25] 
 org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Reporting fetch 
 failure for attempt_1352354913026_138167_m_000654_0 to jobtracker.
 {noformat}
 Even though the error was local to the reducer, it reported the error as a 
 fetch failure to the AM than failing the reducer itself.  It then proceeded 
 to run into the same error for many other maps, causing them to relaunch from 
 reported fetch failures.  In this case it would have been better to fail the 
 reducer and try another node rather than blame the mapper for what is an 
 error on the reducer's side.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-4816) JobImpl Invalid event: JOB_TASK_ATTEMPT_COMPLETED at FAILED

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-4816.
---

Resolution: Duplicate

Fixed by MAPREDUCE-4774.

 JobImpl Invalid event: JOB_TASK_ATTEMPT_COMPLETED at FAILED
 ---

 Key: MAPREDUCE-4816
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4816
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Affects Versions: 0.23.5
Reporter: Jason Lowe

 Saw this in an AM log of a task that had failed:
 {noformat}
 2012-11-21 23:26:44,533 ERROR [AsyncDispatcher event handler] 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event 
 at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 JOB_TASK_ATTEMPT_COMPLETED at FAILED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
   at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:690)
   at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
   at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:904)
   at 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:900)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
   at java.lang.Thread.run(Thread.java:619)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-3111) Fix log serving in NodeManager

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-3111.
---

Resolution: Duplicate

Fixed by MAPREDUCE-4283.

 Fix log serving in NodeManager
 --

 Key: MAPREDUCE-3111
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3111
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.23.0
Reporter: Luke Lu
Assignee: Omkar Vinit Joshi
  Labels: security, webapp

 Just noticed that the current log serving is using the raw writer (instead of 
 Hamlet) to serve logs without escaping html.
 It's actually easier/cleaner to use Hamlet to serve logs:
 {code}
 pre._(buffer);
 {code}
 which takes care of content escaping automatically.
 I will make raw writer access package private for framework use only.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (MAPREDUCE-3111) Fix log serving in NodeManager

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-3111:
---


This was fixed by MAPREDUCE-4283, as that patch started using Hamlet's pre 
method to emit the log data.  Reopening to close this as a duplicate for proper 
tracking of when this was fixed.

 Fix log serving in NodeManager
 --

 Key: MAPREDUCE-3111
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3111
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.23.0
Reporter: Luke Lu
Assignee: Omkar Vinit Joshi
  Labels: security, webapp

 Just noticed that the current log serving is using the raw writer (instead of 
 Hamlet) to serve logs without escaping html.
 It's actually easier/cleaner to use Hamlet to serve logs:
 {code}
 pre._(buffer);
 {code}
 which takes care of content escaping automatically.
 I will make raw writer access package private for framework use only.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5792) When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5792.
---

Resolution: Duplicate

Thanks, Travis.  Resolving as a duplicate of YARN-675.

 When mapreduce.jobhistory.intermediate-done-dir isn't writable, application 
 fails with generic error
 

 Key: MAPREDUCE-5792
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5792
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Travis Thompson
Assignee: Mohammad Kamrul Islam

 When trying to run an application and the permissions are wrong on 
 {{mapreduce.jobhistory.intermediate-done-dir}}, the MapReduce AM fails with a 
 non-descriptive error message:
 {noformat}
 Application application_1394227890066_0004 failed 2 times due to AM Container 
 for appattempt_1394227890066_0004_02 exited with exitCode: 1 due to: 
 Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
 at org.apache.hadoop.util.Shell.run(Shell.java:418)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
 at 
 org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 main : command provided 1
 main : user is tthompso
 main : requested yarn user is tthompso
 Container exited with a non-zero exit code 1
 .Failing this attempt.. Failing the application. 
 {noformat}
 When permissions are corrected on this dir, applications are able to run.  
 There should probably be some sort of check on this dir before launching the 
 AM so a more meaningful error message can be thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-5778) JobSummary does not escape newlines in the job name

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5778:
-

 Summary: JobSummary does not escape newlines in the job name
 Key: MAPREDUCE-5778
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5778
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.3.0, 0.23.10
Reporter: Jason Lowe


JobSummary is not escaping newlines in the job name.  This can result in a job 
summary log entry that spans multiple lines when users are expecting 
one-job-per-line output.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (MAPREDUCE-5764) Potential NullPointerException in YARNRunner.killJob(JobID arg0)

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5764.
---

Resolution: Duplicate

This is a duplicate of MAPREDUCE-5542.

 Potential NullPointerException in YARNRunner.killJob(JobID arg0)
 

 Key: MAPREDUCE-5764
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5764
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Rohith
Assignee: Rohith

 I found YARNRunner.killJob(JobID arg0) can throw NullPointerExpetion if job 
 status is null. 
 bq. clientCache.getClient(arg0).getJobStatus(arg0);  can be null.
 This can happen when there is history write is failed because of hdfs errors 
 or staging directory is different from history server..
  
 We need to have null check otherwise killJob() is prone to throw NPE which 
 cause joblient to exit.
 {noformat}
 @Override
   public void killJob(JobID arg0) throws IOException, InterruptedException {
 /* check if the status is not running, if not send kill to RM */
 JobStatus status = clientCache.getClient(arg0).getJobStatus(arg0);
 if (status.getState() != JobStatus.State.RUNNING) {
   try {
 resMgrDelegate.killApplication(TypeConverter.toYarn(arg0).getAppId());
   } catch (YarnException e) {
 throw new IOException(e);
   }
   return;
 }
 ...
 ..
 ...
   }
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (MAPREDUCE-5758) Reducer local data is not deleted until job completes

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5758:
-

 Summary: Reducer local data is not deleted until job completes
 Key: MAPREDUCE-5758
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5758
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.2.0, 0.23.10
Reporter: Jason Lowe


Ran into an instance where a reducer shuffled a large amount of data and 
subsequently failed, but the local data is not purged when the task fails but 
only after the entire job completes.  This wastes disk space unnecessarily 
since the data is no longer relevant after the task-attempt exits.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (MAPREDUCE-5746) Job diagnostics can implicate wrong task for a failed job

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5746:
-

 Summary: Job diagnostics can implicate wrong task for a failed job
 Key: MAPREDUCE-5746
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5746
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.1.1-beta, 0.23.10
Reporter: Jason Lowe


We've seen a number of cases where the history server is showing the wrong task 
as the reason a job failed.  For example, Task 
task_1383802699973_515536_m_027135 failed 1 times when some other task had 
failed 4 times and was the real reason the job failed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5584.
---

Resolution: Duplicate

This has been fixed by the addendum patch in HADOOP-9652.

 ShuffleHandler becomes unresponsive during gridmix runs and can leak file 
 descriptors
 -

 Key: MAPREDUCE-5584
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Jason Lowe
Priority: Blocker

 While running gridmix on 2.3 we noticed that jobs are running much slower 
 than normal.  We tracked this down to reducers having difficulties shuffling 
 data from maps.  Details to follow.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (MAPREDUCE-5730) AM log is truncated

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5730.
---

Resolution: Duplicate

Fixing this as part of MAPREDUCE-5723.

 AM log is truncated
 ---

 Key: MAPREDUCE-5730
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5730
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.4.0
Reporter: Jason Lowe
Priority: Critical

 The ApplicationMaster log is being truncated with only the log messages up 
 until around the point where the job is being setup are present.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (MAPREDUCE-5730) AM log is truncated

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5730:
-

 Summary: AM log is truncated
 Key: MAPREDUCE-5730
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5730
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.4.0
Reporter: Jason Lowe
Priority: Critical


The ApplicationMaster log is being truncated with only the log messages up 
until around the point where the job is being setup are present.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (MAPREDUCE-5727) History server web page can filter without showing filter keyword

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5727:
-

 Summary: History server web page can filter without showing filter 
keyword
 Key: MAPREDUCE-5727
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5727
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.4.0
Reporter: Jason Lowe


I loaded up a job conf page on the history server and used one of the search 
boxes to narrow the results.  I then navigated to other pages (e.g.: map tasks, 
logs, etc.) then navigated back to the job conf page using the job 
configuration link on the left side of the page.  When I arrived it promptly 
showed me just a few conf entries (the ones I had searched for earlier) but my 
search term was missing.  At first glance it looked like those were the only 
entries in the entire job conf, which can be very confusing.  Somehow the 
search term is being remembered but not replotted when the configuration page 
is revisited.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (MAPREDUCE-5686) Found Class org.apache.hadoop.mapreduce.TaskAttemptContext,but interface was excepted

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5686.
---

Resolution: Invalid

As pointed out on the other JIRAs you have filed, _please_ ask these kinds of 
questions on the [user@ mailing 
list|http://hadoop.apache.org/mailing_lists.html#User].  JIRA is for Hadoop 
developers to track bugs and is not a channel for general user support.  That 
is why the user@ mailing list exists.

The problem here is the same type of problem you reported in MAPREDUCE-5666 and 
MAPREDUCE-5668.  It looks like you are compiling against a later version of 
Hadoop than you are running on, and that is not supported.

 Found Class org.apache.hadoop.mapreduce.TaskAttemptContext,but interface was 
 excepted
 -

 Key: MAPREDUCE-5686
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5686
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: ranjini

 hi,
 Iam using the hadoop version 0.20. 
 Please suggest to fix the bug.
 Thanks in advance.
 Ranjini



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (MAPREDUCE-5635) FileInputFormat does not specify how the file is split

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5635.
---

Resolution: Not A Problem

Closing this as FileInputFormat is not supposed to specify the specifics on the 
file format per the previous comment.

 FileInputFormat does not specify how the file is split
 --

 Key: MAPREDUCE-5635
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5635
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.2.0
 Environment: Does not matter.
Reporter: Pranay Varma

 Here is what the TextInputFormat javadoc says:
 [TextInputFormat|http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html]
 An InputFormat for plain text files. Files are broken into lines. Either 
 linefeed or carriage-return are used to signal end of line. Keys are the 
 position in the file, and values are the line of text..
 FileInputFormat should say the same on
 [FileInputFormat|http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5670) CombineFileRecordReader should report progress when moving to the next file

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5670:
-

 Summary: CombineFileRecordReader should report progress when 
moving to the next file
 Key: MAPREDUCE-5670
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5670
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.2.0, 0.23.9
Reporter: Jason Lowe
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (MAPREDUCE-5661) ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5661.
---

Resolution: Not A Problem

Closing as shufflehandler should continue to use the yarn property since it's 
part of the nodemanager process.

 ShuffleHandler using yarn.nodemanager.local-dirs instead of 
 mapreduce.cluster.local.dir
 ---

 Key: MAPREDUCE-5661
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5661
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Eric Sirianni
Priority: Trivial

 While debugging an issue where a MapReduce job is failing due to running out 
 of disk space, I noticed that the {{ShuffleHandler}} uses 
 {{yarn.nodemanager.local-dirs}} for its {{LocalDirAllocator}} whereas all of 
 the other MapReduce classes use {{mapreduce.cluster.local.dir}}:
 {noformat}
 $ find hadoop-mapreduce-project/hadoop-mapreduce-client/*/src/main/java/ 
 -name *.java | xargs grep new LocalDirAllocator(
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java:
 LocalDirAllocator lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java:
 new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java:
   new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java:
   this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java:
 new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java:
 new LocalDirAllocator(MRConfig.LOCAL_DIR);
 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:
 this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
 *hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java:
   new LocalDirAllocator(YarnConfiguration.NM_LOCAL_DIRS);
 {noformat}
 This inconsistency feels like something that is likely to confuse admins.  



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (MAPREDUCE-5666) org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java(org/apache/hadoop/mapreduce/lib/input:FileInputFormat.java):cannot find symbol

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5666.
---

Resolution: Invalid

As I mentioned in MAPREDUCE-5668, JIRA is not the avenue to use for questions 
like this.  Please use the [user@ mailing 
list|http://hadoop.apache.org/mailing_lists.html#User] for questions like this. 
 If after discussing on the mailing list it ends up being a bug in Hadoop then 
a JIRA can be filed at that time.

This is clearly a case of code being compiled against a release after 1.x but 
then run on a 0.20 or 1.x release.  The FileStatus.isDirectory() method was not 
present in the 0.20 or 1.x releases.  We cannot generally support compiling 
code against a later release and then running it on an earlier release because 
of new APIs that can be added.

 org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java(org/apache/hadoop/mapreduce/lib/input:FileInputFormat.java):cannot
  find symbol
 -

 Key: MAPREDUCE-5666
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5666
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: ranjini

 hi 
 I have written the below code , and facing the issue. i am using hadoop 0.20 
 vesion and java 1.6 the issue is 
 org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java(org/apache/hadoop/mapreduce/lib/input:FileInputFormat.java):232:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
   if (globStat.isDirectory()) {
   ^
 org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java(org/apache/hadoop/mapreduce/lib/output:FileOutputCommitter.java):208:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
 } else if(fs.getFileStatus(taskOutput).isDirectory()) {
   ^
 org/apache/hadoop/mapred/JobConf.java(org/apache/hadoop/mapred:JobConf.java):433:
  cannot find symbol
 symbol  : method getPattern(java.lang.String,java.util.regex.Pattern)
 location: class org.apache.hadoop.mapred.JobConf
 return getPattern(JobContext.JAR_UNPACK_PATTERN, 
 UNPACK_JAR_PATTERN_DEFAULT);
^
 org/apache/hadoop/mapred/JobConf.java(org/apache/hadoop/mapred:JobConf.java):450:
  cannot find symbol
 symbol  : method getTrimmedStrings(java.lang.String)
 location: class org.apache.hadoop.mapred.JobConf
 return getTrimmedStrings(MRConfig.LOCAL_DIR);
^
 org/apache/hadoop/mapred/FileInputFormat.java(org/apache/hadoop/mapred:FileInputFormat.java):165:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
   if (stat.isDirectory()) {
   ^
 org/apache/hadoop/mapred/FileInputFormat.java(org/apache/hadoop/mapred:FileInputFormat.java):215:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
   if (globStat.isDirectory()) {
   ^
 org/apache/hadoop/mapred/FileInputFormat.java(org/apache/hadoop/mapred:FileInputFormat.java):218:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
   if (recursive  stat.isDirectory()) {
^
 org/apache/hadoop/mapred/FileInputFormat.java(org/apache/hadoop/mapred:FileInputFormat.java):258:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
   if (file.isDirectory()) {
   ^
 org/apache/hadoop/mapred/FileOutputCommitter.java(org/apache/hadoop/mapred:FileOutputCommitter.java):166:
  cannot find symbol
 symbol  : method isDirectory()
 location: class org.apache.hadoop.fs.FileStatus
 } else if(fs.getFileStatus(taskOutput).isDirectory()) {
   ^
 org/apache/hadoop/mapred/LineRecordReader.java(org/apache/hadoop/mapred:LineRecordReader.java):100:
  incompatible types
 found   : org.apache.hadoop.io.compress.SplitCompressionInputStream
 required: org.apache.hadoop.fs.Seekable
 filePosition = cIn; // take pos from compressed stream
^
 org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java(org/apache/hadoop/mapreduce/lib/input:LineRecordReader.java):98:
  incompatible types
 found   : org.apache.hadoop.io.compress.SplitCompressionInputStream
 required: org.apache.hadoop.fs.Seekable
 filePosition = cIn;
 I have attached the code 
 import java.io.DataInput;  
 import java.io.DataOutput;  
 import java.io.IOException;  
 import java.util.StringTokenizer;  
 import org.apache.hadoop.conf.Configured;  
 import org.apache.hadoop.fs.FSDataInputStream;  
 import 

[jira] [Resolved] (MAPREDUCE-5668) Exception in thread main java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.JobContext, but interface was expected

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5668.
---

Resolution: Invalid

Closing this as invalid based on the evidence from MAPREDUCE-5667.  The code is 
being compiled against a later release but then run on an earlier release.  The 
code should be compiled against the Hadoop release being used or an earlier 
release, keeping in mind the binary compatibility document guidelines.

 Exception in thread main java.lang.IncompatibleClassChangeError: Found 
 class org.apache.hadoop.mapreduce.JobContext, but interface was expected
 -

 Key: MAPREDUCE-5668
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5668
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: ranjini

 hi
  pl help
 i have wrote this code , at runtime i got this issue.
 Exception in thread main java.lang.IncompatibleClassChangeError: Found 
 class org.apache.hadoop.mapreduce.JobContext, but interface was expected
   at 
 org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:170)
   at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
   at 
 org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
   at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
   at MultiFileWordCount.run(MultiFileWordCount.java:395)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
   at MultiFileWordCount.main(MultiFileWordCount.java:401)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
 hduser@localhost:~$ 
 I have attached the code.
 import java.io.DataInput;  
 import java.io.DataOutput;  
 import java.io.IOException;  
 import java.util.StringTokenizer;  
 import org.apache.hadoop.conf.Configured;  
 import org.apache.hadoop.fs.FSDataInputStream;  
 import org.apache.hadoop.fs.FileSystem;  
 import org.apache.hadoop.fs.Path;  
 import org.apache.hadoop.io.IntWritable;  
 import org.apache.hadoop.io.Text;  
 import org.apache.hadoop.io.WritableComparable;  
 import org.apache.hadoop.mapreduce.InputSplit;  
 import org.apache.hadoop.mapreduce.Job;  
 import org.apache.hadoop.mapreduce.Mapper;  
 import org.apache.hadoop.mapreduce.RecordReader;  
 import org.apache.hadoop.mapreduce.TaskAttemptContext;  
 import org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat;  
 import org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader;  
 import org.apache.hadoop.mapreduce.lib.input.CombineFileSplit;  
 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
 import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer;  
 import org.apache.hadoop.util.LineReader;  
 import org.apache.hadoop.util.Tool;  
 import org.apache.hadoop.util.ToolRunner;  
  /**  
   * MultiFileWordCount is an example to demonstrate the usage of   
   * MultiFileInputFormat. This examples counts the occurrences of  
   * words in the text files under the given input directory.  
   */ 
 public class MultiFileWordCount extends Configured implements Tool {  
/**  
 * This record keeps filename,offset pairs.  
 */ 
 public static class WordOffset implements WritableComparable {  
private long offset;  
private String fileName;  
   
public void readFields(DataInput in) throws IOException {  
   this.offset = in.readLong();  
   this.fileName = Text.readString(in);  
  }  
  public void write(DataOutput out) throws IOException {  
out.writeLong(offset);  
Text.writeString(out, fileName);  
  }  
   public int compareTo(Object o) {  
WordOffset that = (WordOffset)o;  
int f = this.fileName.compareTo(that.fileName);  
if(f == 0) {  
  return (int)Math.signum((double)(this.offset - that.offset));  
}  
return f;  
  }  
  @Override 
  public boolean equals(Object obj) {  
if(obj instanceof WordOffset)  
return this.compareTo(obj) == 0;  
return false;  
  }  
  @Override 
  public int hashCode() {  
  assert false : hashCode not designed;  
  return 42; //an arbitrary constant  
  }  
}  
   

[jira] [Created] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5584:
-

 Summary: ShuffleHandler becomes unresponsive during gridmix runs 
and can leak file descriptors
 Key: MAPREDUCE-5584
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Jason Lowe


While running gridmix on 2.3 we noticed that jobs are running much slower than 
normal.  We tracked this down to reducers having difficulties shuffling data 
from maps.  Details to follow.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (MAPREDUCE-5581) killing jobs which have failed causes log missing

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5581.
---

Resolution: Duplicate

This is a duplicate of MAPREDUCE-5502.

 killing jobs which have failed causes log missing
 -

 Key: MAPREDUCE-5581
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5581
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.1.1-beta
Reporter: Nemon Lou

 In hive code,when a job failed,they invoke the RunningJob.killJob() API 
 immediately.
 From mapreduce client side,when job is at failed state,the YARNRunner will 
 invoke resMgrDelegate.killApplication to kill that job.And this prevent AM 
 from writing logs to job history server.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5583) Ability to limit running map and reduce tasks

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5583:
-

 Summary: Ability to limit running map and reduce tasks
 Key: MAPREDUCE-5583
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mr-am, mrv2
Affects Versions: 2.1.1-beta, 0.23.9
Reporter: Jason Lowe


It would be nice if users could specify a limit to the number of map or reduce 
tasks that are running simultaneously.  Occasionally users are performing 
operations in tasks that can lead to DDoS scenarios if too many tasks run 
simultaneously (e.g.: accessing a database, web service, etc.).  Having the 
ability to throttle the number of tasks simultaneously running would provide 
users a way to mitigate issues with too many tasks on a large cluster 
attempting to access a serivce at any one time.

This is similar to the functionality requested by MAPREDUCE-224 and implemented 
by HADOOP-3412 but was dropped in mrv2.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5574) History server returns 500 error on job conf page if user lacks permissions

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5574:
-

 Summary: History server returns 500 error on job conf page if user 
lacks permissions
 Key: MAPREDUCE-5574
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5574
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.1.1-beta, 0.23.9
Reporter: Jason Lowe


If a user does not have view ACL permissions for a job and tries to view the 
job configuration URL (i.e.: .../jobhistory/conf/jobid) then the history 
server returns a 500 error rather than a descriptive error message informing 
the user that they lack permissions.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5570:
-

 Summary: Map task attempt with fetch failure has incorrect attempt 
finish time
 Key: MAPREDUCE-5570
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.1.1-beta, 0.23.9
Reporter: Jason Lowe


If a map task attempt is retroactively failed due to excessive fetch failures 
reported by reducers then the attempt's finish time is set to the time the task 
was retroactively failed rather than when the task attempt completed.  This 
causes the map task attempt to appear to have run for much longer than it 
actually did.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5542) Killing a job just as it finishes can generate an NPE in client

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5542:
-

 Summary: Killing a job just as it finishes can generate an NPE in 
client
 Key: MAPREDUCE-5542
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5542
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe


If a client tries to kill a job just as the job is finishing then the client 
can crash with an NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5543) In-memory map outputs can be leaked after shuffle completes in 0.23

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5543:
-

 Summary: In-memory map outputs can be leaked after shuffle 
completes in 0.23
 Key: MAPREDUCE-5543
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5543
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.1.0-beta, 0.23.9
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.1.1-beta


MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and 
inMemoryMapOutputs to a list of map outputs that is subsequently processed, but 
it does not clear those sets.  This prevents some of the map outputs from being 
garbage collected and significantly reduces the memory available for the 
subsequent reduce phase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5513) ConcurrentModificationException in JobControl

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5513:
-

 Summary: ConcurrentModificationException in JobControl
 Key: MAPREDUCE-5513
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5513
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.9, 2.1.0-beta
Reporter: Jason Lowe


JobControl.toList is locking individual lists to iterate them, but those lists 
can be modified elsewhere without holding the list lock.  The locking 
approaches are mismatched, with toList holding the lock on the actual list 
object while other methods hold the JobControl lock when modifying the lists.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5503) TestMRJobClient.testJobClient is failing

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5503:
-

 Summary: TestMRJobClient.testJobClient is failing
 Key: MAPREDUCE-5503
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5503
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0
Reporter: Jason Lowe


TestMRJobClient.testJobClient is failing on trunk and causing precommit builds 
to complain:

{noformat}
testJobClient(org.apache.hadoop.mapreduce.TestMRJobClient)  Time elapsed: 
26.361 sec   FAILURE!
junit.framework.AssertionFailedError: expected:1 but was:0
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at 
org.apache.hadoop.mapreduce.TestMRJobClient.testJobList(TestMRJobClient.java:474)
at 
org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:112)
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5475) MRClientService does not verify ACLs properly

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5475.
---

   Resolution: Fixed
Fix Version/s: 0.23.10

I committed this to trunk, branch-2, branch-2.1-beta, and branch-0.23.

 MRClientService does not verify ACLs properly
 -

 Key: MAPREDUCE-5475
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.0.4-alpha, 0.23.9
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 0.23.10, 2.1.1-beta

 Attachments: MAPREDUCE-5475.branch-0.23.patch, MAPREDUCE-5475.patch


 When MRClientService receives requests, it calls verifyAndGetJob which does 
 not actually validate that the current user has the proper access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5493) In-memory map outputs can be leaked after shuffle completes

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5493:
-

 Summary: In-memory map outputs can be leaked after shuffle 
completes
 Key: MAPREDUCE-5493
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5493
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.9, 2.1.0-beta
Reporter: Jason Lowe


MergeManagerImpl#close adds the contents of inMemoryMergedMapOutputs and 
inMemoryMapOutputs to a list of map outputs that is subsequently processed, but 
it does not clear those sets.  This prevents some of the map outputs from being 
garbage collected and significantly reduces the memory available for the 
subsequent reduce phase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5481) TestUberAM timeout

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5481:
-

 Summary: TestUberAM timeout
 Key: MAPREDUCE-5481
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 3.0.0
Reporter: Jason Lowe


TestUberAM has been timing out on trunk for some time now and surefire then 
fails the build.  I'm not able to reproduce it locally, but the Jenkins builds 
have been seeing it fairly consistently.  See 
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5475) MRClientService does not verify ACLs properly

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5475.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Verified that the original patch works on a secure cluster once YARN-707 was 
committed, so I committed this again to trunk, branch-2, and branch-2.1-beta.

 MRClientService does not verify ACLs properly
 -

 Key: MAPREDUCE-5475
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.0.4-alpha, 0.23.9
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.1.1-beta

 Attachments: MAPREDUCE-5475.branch-0.23.patch, MAPREDUCE-5475.patch


 When MRClientService receives requests, it calls verifyAndGetJob which does 
 not actually validate that the current user has the proper access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-5475) MRClientService does not verify ACLs properly

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-5475:
---


Reverting this again... With YARN-707 users can see their own jobs again, but 
other users can still kill them because the token user is always the app 
submitter, and the AM sees all authenticated client connections coming from 
that user.

 MRClientService does not verify ACLs properly
 -

 Key: MAPREDUCE-5475
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.0.4-alpha, 0.23.9
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.1.1-beta

 Attachments: MAPREDUCE-5475.branch-0.23.patch, MAPREDUCE-5475.patch


 When MRClientService receives requests, it calls verifyAndGetJob which does 
 not actually validate that the current user has the proper access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-5475) MRClientService does not verify ACLs properly

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-5475:
---


Reopening, as this appears to be causing problems on a secure cluster.  Getting 
exceptions on the job submission client after the job starts running and it 
tries to connect to monitor the job:

13/08/22 18:51:59 ERROR security.UserGroupInformation:
PriviledgedActionException as:x@y (auth:KERBEROS)
cause:java.io.IOException: org.apache.hadoop.security.AccessControlException:
User appattempt_1377189855273_0006_01 cannot perform operation VIEW_JOB

Will revert the change for now until we better understand what's going on.

 MRClientService does not verify ACLs properly
 -

 Key: MAPREDUCE-5475
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 2.0.4-alpha, 0.23.9
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.1.1-beta

 Attachments: MAPREDUCE-5475.branch-0.23.patch, MAPREDUCE-5475.patch


 When MRClientService receives requests, it calls verifyAndGetJob which does 
 not actually validate that the current user has the proper access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5469) Counters for MRAppMaster

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5469.
---

Resolution: Duplicate

Duplicate of MAPREDUCE-4804.

 Counters for MRAppMaster
 

 Key: MAPREDUCE-5469
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5469
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.0.5-alpha
Reporter: nemon lou

 We have counters for map tasks and reduce tasks ,but has no counters for 
 MRAppMaster.
 Sometimes we need information like GC time,memory usage for AM tuning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5456) TestFetcher.testCopyFromHostExtraBytes is missing

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5456:
-

 Summary: TestFetcher.testCopyFromHostExtraBytes is missing
 Key: MAPREDUCE-5456
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5456
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.1.0-beta
Reporter: Jason Lowe
Priority: Minor


I noticed that the test to verify the fix from MAPREDUCE-5308 was deleted by 
MAPREDUCE-5194.  It looks like an accidental deletion from an upmerge.  
We should reinstate this unit test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5444) MRAppMaster throws InvalidStateTransitonException: Invalid event: JOB_AM_REBOOT at SUCCEEDED

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-5444.
---

Resolution: Invalid

bq. I have one point to add here that, immidiately after job is succeeded , app 
master got reboot command from RM. JobClient is exitted( see MAPREDUCE-5441 ). 
By the time, RM has launched 2nd attempt of app master. 2nd attempt app master 
too compete for resources, but there is no client waiting getting job report.I 
feel this is problem.

There will always be a race where the job has just succeeded but the RM gets 
out of sync with the AM before the AM can unregister.  Normally the AM will 
exit, another AM attempt will be launched by the RM, and the new attempt will 
recover the previous SUCCEEDED state and exit shortly afterwards without 
launching any subsequent tasks.

As for the client, that's an orthogonal problem.  It's not required that a 
client be listening to an application as it executes, and if the client is 
unnecessarily exiting across an AM restart then we can tackle that issue in 
MAPREDUCE-5441.

 MRAppMaster throws InvalidStateTransitonException: Invalid event: 
 JOB_AM_REBOOT at SUCCEEDED
 

 Key: MAPREDUCE-5444
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5444
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: applicationmaster
Reporter: Rohith Sharma K S
Priority: Minor

 {noformat}
 2013-08-02 14:55:11,537 INFO [AsyncDispatcher event handler] 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for 
 JobFinishedEvent 
 2013-08-02 14:55:11,538 INFO [AsyncDispatcher event handler] 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: 
 job_1375199817609_0049Job Transitioned from COMMITTING to SUCCEEDED
 2013-08-02 14:55:11,663 INFO [Thread-52] 
 org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying 
 hdfs://0.0.0.0:45000/home/restest/staging-dir/restest/.staging/job_1375199817609_0049/job_1375199817609_0049_2.jhist
  to 
 hdfs://0.0.0.0:45000/home/restest/staging-dir/history/done_intermediate/restest/job_1375199817609_0049-1375435337429-restest-word+count-1375435511533-10-1-SUCCEEDED-a.jhist_tmp
 2013-08-02 14:55:11,750 INFO [Thread-52] 
 org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done 
 location: 
 hdfs://0.0.0.0:45000/home/restest/staging-dir/history/done_intermediate/restest/job_1375199817609_0049-1375435337429-restest-word+count-1375435511533-10-1-SUCCEEDED-a.jhist_tmp
 2013-08-02 14:55:11,769 INFO [Thread-52] 
 org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying 
 hdfs://0.0.0.0:45000/home/restest/staging-dir/restest/.staging/job_1375199817609_0049/job_1375199817609_0049_2_conf.xml
  to 
 hdfs://0.0.0.0:45000/home/restest/staging-dir/history/done_intermediate/restest/job_1375199817609_0049_conf.xml_tmp
 2013-08-02 14:55:11,880 INFO [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before 
 Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 
 AssignedReds:1 CompletedMaps:10 CompletedReds:1 ContAlloc:1 ContRel:0 
 HostLocal:0 RackLocal:0
 2013-08-02 14:55:13,649 ERROR [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Error 
 communicating with RM: Resource Manager doesn't recognize AttemptId: 
 application_1375199817609_0049
 org.apache.hadoop.yarn.YarnException: Resource Manager doesn't recognize 
 AttemptId: application_1375199817609_0049
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:626)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:238)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:250)
   at java.lang.Thread.run(Thread.java:662)
 2013-08-02 14:55:13,649 ERROR [AsyncDispatcher event handler] 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event 
 at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 JOB_AM_REBOOT at SUCCEEDED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
   at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:914)
   at 
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:129)
   at 
 

[jira] [Created] (MAPREDUCE-5445) MRApp tries to stop services from AsyncDispatcher thread

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-5445:
-

 Summary: MRApp tries to stop services from AsyncDispatcher thread
 Key: MAPREDUCE-5445
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5445
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.5-alpha
Reporter: Jason Lowe


MRApp tries to stop services from within a service thread.  That causes the 
stop call itself to be interrupted and not all services are stopped.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira