[
https://issues.apache.org/jira/browse/HADOOP-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HADOOP-4261:
--------------------------------------------
Attachment: patch-4261.txt
Attaching patch thats modifies committer.setupJob() as a separate task similar
to cleanupTask.
* With the patch, JobInProgress.initTasks() initializes all the tips (including
setup and cleanup tips). SetupTask will be launched when the job is in PREP
state, after the tasks get initialized. Once the setupTask completes, the Job
is moved to RUNNING state.
* JobTracker first iterates over the jobs for cleanup tasks, then setup tasks.
If there are no cleanup and setup tasks, it calls taskScheduler.assignTasks()
for maps and reduces.
* This task is logged in JobHistory with SETUP as TASK_TYPE.
* The Job web UI and History web UI is modified to add setup task. The UI
change is similar to the cleanup task UI.
* Updated Javadoc and documentation
> Jobs failing in the init stage will never cleanup
> -------------------------------------------------
>
> Key: HADOOP-4261
> URL: https://issues.apache.org/jira/browse/HADOOP-4261
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amar Kamat
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: patch-4261.txt
>
>
> Pre HADOOP-3150, if the job fails in the init stage, {{job.kill()}} was
> called. This used to make sure that the job was cleaned up w.r.t
> - staus set to KILLED/FAILED
> - job files from the system dir are deleted
> - closing of job history files
> - making jobtracker aware of this through {{jobTracker.finalizeJob()}}
> - cleaning up the data structures via {{JobInProgress.garbageCollect()}}
> Now if the job fails in the init stage, {{job.fail()}} is called which doesnt
> do the cleanup. HADOOP-3150 introduces cleanup tasks which are launched once
> the job completes i.e killed/failed/succeeded. Jobtracker will never
> consider this job for scheduling as the job will be in the {{PREP}} state
> forever.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
