[
https://issues.apache.org/jira/browse/HADOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584183#action_12584183
]
Amar Kamat commented on HADOOP-3140:
------------------------------------
How about this
1) {{Task.done()}} method checks if the task has data to be promoted and passes
this info to the TaskTracker via the {{TaskTracker.done()}} api.
2) If there is no data to promote, the TaskTracker sets the task status as
{{SUCCEEDED}} or {{FAILED}} depending on whether the task succeeds or fails.
3) JobInProgress adds only {{COMMIT_PENDING}} tasks to the commit-pending
queue. The commit-pending queue deals with {{KILLED/FAILED}} tasks only if the
commit-pending thread fails to save the task output or if the TaskTracker is
lost.
4) Temporary data from {{FAILED/KILLED}} tasks will be deleted once the job
completes (see HADOOP-2391).
5) {{JobInProgress.updateTaskStatus()}} can now be called with {{SUCCEEDED}}
state from TaskTracker (via heartbeat) or from the commit-pending queue.
5) If a JobInProgress.updateTaskStatus() is called with {{SUCCEEDED}} state for
a completed TIP it will be marked as {{KILLED}}.
> JobTracker should not try to promote a (map) task if it does not write to DFS
> at all
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3140
> URL: https://issues.apache.org/jira/browse/HADOOP-3140
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> In most cases, map tasks do not write to dfs.
> Thus, when they complete, they should not be put into commit_pending queue at
> all.
> This will improve the task promotion significantly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
