[
https://issues.apache.org/jira/browse/HADOOP-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603772#action_12603772
]
Amar Kamat commented on HADOOP-3245:
------------------------------------
There is a deficiency with the _SYNC_ algo mentioned above. The _task
completion events_ in the JT will now be reordered. The copy of _task
completion events_ that the TT has is now garbled. Consider the following cases
with the SYNC algo in place
1) A TT has reducers in _SHUFFLE_ phase : Here the TT will have a partial copy
of the previous version of the _task completion events_ and hence will require
the latest copy.
2) A TT has all the reducers in _REDUCE_ phase before the JT restart and new
reduce tasks assigned to it after the restart : Here the TT will have a
complete copy of the previous version of the _task completion events_ and hence
it looks like it might not require the latest copy of _completion events_. But
consider a case where some maps were lost and hence their output is not
available. Since the ordering of the _task completion events_ is different and
some events in the JT might belong to re-executed maps, there is no good way to
inform the TTs that a particular map was lost and it should use the new _task
completion event_. Currently (with trunk) this is not a issue because there
will be just one copy of the _task completion events_ in the lifetime of the
job. Hence the TT will always have the correct ordering of the _completion
events_. In case of re-executions the new event will always be at the end.
----
One simple solution would be to force the re-build of _completion events_ copy
at the TT on _SYNC_ action.
> Provide ability to persist running jobs (extend HADOOP-1876)
> ------------------------------------------------------------
>
> Key: HADOOP-3245
> URL: https://issues.apache.org/jira/browse/HADOOP-3245
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Amar Kamat
>
> This could probably extend the work done in HADOOP-1876. This feature can be
> applied for things like jobs being able to survive jobtracker restarts.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
