[
https://issues.apache.org/jira/browse/MAPREDUCE-4748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4748:
----------------------------------
Attachment: MAPREDUCE-4748.patch
Simple patch to ignore T_ATTEMPT_SUCCEEDED, T_KILL, and
T_ATTEMPT_COMMIT_PENDING at SUCCEEDED and keep the job from abruptly ending in
error.
I'm a bit worried about the bookkeeping wrt. task.finishedAttempts and
task.numberUncompletedAttempts. Current patch matches the bookkeeping behavior
for T_ATTEMPT_KILLED or T_ATTEMPT_FAILED when we're effectively ignoring the
event. However I'm wondering if this could lead to corner cases during
KILL_WAIT like those reported in MAPREDUCE-4745.
It looks like TaskAttempt will report T_ATTEMPT_KILLED after it succeeded but
only for map tasks. We don't want to double-count in that case, but if a kill
of the TaskAttempt doesn't report it was killed it seems like we could miss
some bookeeping if we just ignore bookkeeping when we see an attempt
redundantly succeeded. Thoughts?
> Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED
> -----------------------------------------------
>
> Key: MAPREDUCE-4748
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4748
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-4748.patch
>
>
> We saw this happen when running a large pig script.
> {noformat}
> 2012-10-23 22:45:24,986 ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Can't handle this event
> at current state for task_1350837501057_21978_m_040453
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> T_ATTEMPT_SUCCEEDED at SUCCEEDED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:604)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:89)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:914)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:908)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> Speculative execution was enabled, and that task did speculate so it looks
> like this is an error in the state machine either between the task attempts
> or just within that single task.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira