[
https://issues.apache.org/jira/browse/MAPREDUCE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459808#comment-16459808
]
Jim Brennan commented on MAPREDUCE-6948:
----------------------------------------
[~jlowe], [~haibochen], I tried to reproduce this again in branch-2.7 and
branch-3.0. I ran this test in a loop for more than an hour on each, and was
not able to get it to fail.
In branch-2.7, I can get it to fail as reported by inserting a sleep into the
taskAttemptEventHandler right before it decrements the succeededMapperCount().
If I do the same thing in branch-3.0, it does not fail (due to the
aforementioned fixes in [MAPREDUCE-6675]).
I searched for this test in apache issues, and the only reference I found was
this comment by [~haibochen] in [MAPREDUCE-6937]:
{quote}
+1 on the branch-2.7-v5 patch. I have filed MAPREDUCE-6948 for the unit test
failure TestJobImpl.testUnusableNodeTransition.
{quote}
That comment looks like it is in response to branch-2.7 test failures.
hadoop.mapreduce.v2.app.job.impl.TestJobImpl is only listed as failing in the
branch-2.7 tests.
[~haibochen], did you reproduce the failure on 3.0.0-alpha4? Is it possible
this was intended to be filed against branch 2.7 as a result of the test
failures from [MAPREDUCE-6937]?
The race does still appear to exist on branch 2.7, so we may want to consider
pulling [MAPREDUCE-6675] back to that branch.
> TestJobImpl.testUnusableNodeTransition failed
> ---------------------------------------------
>
> Key: MAPREDUCE-6948
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6948
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha4
> Reporter: Haibo Chen
> Assignee: Jim Brennan
> Priority: Major
> Labels: unit-test
>
> *Error Message*
> expected:<SUCCEEDED> but was:<ERROR>
> *Stacktrace*
> java.lang.AssertionError: expected:<SUCCEEDED> but was:<ERROR>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:1041)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:615)
> *Standard out*
> {code}
> 2017-08-30 10:12:21,928 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2017-08-30 10:12:21,939 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$StubbedJob
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.jobhistory.EventType for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,940 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
> org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$79f96ebf
> 2017-08-30 10:12:21,941 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:setup(1534)) - Adding job token for job_1234567890000_0001 to
> jobTokenSecretManager
> 2017-08-30 10:12:21,941 WARN [Thread-49] impl.JobImpl
> (JobImpl.java:setup(1540)) - Shuffle secret key missing from job credentials.
> Using job token secret as shuffle secret.
> 2017-08-30 10:12:21,944 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:makeUberDecision(1305)) - Not uberizing job_1234567890000_0001
> because: not enabled;
> 2017-08-30 10:12:21,944 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:createMapTasks(1562)) - Input size for job
> job_1234567890000_0001 = 0. Number of splits = 2
> 2017-08-30 10:12:21,945 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:createReduceTasks(1579)) - Number of reduces for job
> job_1234567890000_0001 = 1
> 2017-08-30 10:12:21,945 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_1234567890000_0001Job Transitioned from NEW
> to INITED
> 2017-08-30 10:12:21,946 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_1234567890000_0001Job Transitioned from
> INITED to SETUP
> 2017-08-30 10:12:21,954 INFO [CommitterEvent Processor #0]
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) -
> Processing the event EventType: JOB_SETUP
> 2017-08-30 10:12:21,978 INFO [AsyncDispatcher event handler] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_1234567890000_0001Job Transitioned from
> SETUP to RUNNING
> 2017-08-30 10:12:21,983 INFO [Thread-49] event.AsyncDispatcher
> (AsyncDispatcher.java:register(209)) - Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$5
> 2017-08-30 10:12:22,000 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:transition(1953)) - Num completed Tasks: 1
> 2017-08-30 10:12:22,029 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:transition(1953)) - Num completed Tasks: 2
> 2017-08-30 10:12:22,032 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:actOnUnusableNode(1354)) - TaskAttempt killed because it ran on
> unusable node Mock for NodeId, hashCode: 1280187896.
> AttemptId:attempt_1234567890000_0001_m_000000_0
> 2017-08-30 10:12:22,032 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:transition(1953)) - Num completed Tasks: 3
> 2017-08-30 10:12:22,032 INFO [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_1234567890000_0001Job Transitioned from
> RUNNING to COMMITTING
> 2017-08-30 10:12:22,032 ERROR [Thread-49] impl.JobImpl
> (JobImpl.java:handle(1009)) - Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at COMMITTING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1007)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:609)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> 2017-08-30 10:12:22,035 INFO [AsyncDispatcher event handler] impl.JobImpl
> (JobImpl.java:handle(1017)) - job_1234567890000_0001Job Transitioned from
> COMMITTING to ERROR
> 2017-08-30 10:12:22,039 INFO [CommitterEvent Processor #1]
> commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) -
> Processing the event EventType: JOB_COMMIT
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
