[
https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HADOOP-3150:
--------------------------------------------
Attachment: patch-3150.txt
Here is patch which makes the job cleanup as separate task at the end of the
job.
When all the maps and reduces of the job have finished or the job is killed,
cleanup task will be launched. Once the cleanup task completes, job is marked
SUCCEDED/FAILED.
Cleanup task has the highest priority in TaskScheduler. If a job is still
running and cleanup task can be launched, then scheduler returns a cleanup task
from assignTasks.
And Cleanup task is essentially a mapTask. MapTask.run is modified to do the
cleanup if the task is a cleanupJob. A flag is set, by the JobInProgress, in
the MapTask to know whether it is normal map or a cleanup.
The patch doesnt update any counters for the cleanup task. And web ui also
doesnt show anything about it. We can have a follow up jira which adds a row
for cleanup task in the jobdetails.
Thoughts?
Changes from earlier patch are in JobInProgress, TaskInProgress, MapTask,
JobQueueTaskScheduler and LimitTasksPerJobTaskScheduler
> Move task file promotion into the task
> --------------------------------------
>
> Key: HADOOP-3150
> URL: https://issues.apache.org/jira/browse/HADOOP-3150
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: 3150.patch, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt
>
>
> We need to move the task file promotion from the JobTracker to the Task and
> move it down into the output format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
