[
https://issues.apache.org/jira/browse/MAPREDUCE-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643232#comment-13643232
]
Arun C Murthy commented on MAPREDUCE-5158:
------------------------------------------
Mayank, thanks for the patch!
I like your idea of using JIP.garbageCollect, but the more I look into it, the
more I'm worried about JIP.garbageCollect. This is because it does a whole lot
of other cleanup (metrics, JT.finalizeJob, JobHistory cleanup etc. etc.) which
aren't implemented defensively enough. For e.g. JobHistory.logSubmitted is
called in JIP.initTasks, but JT.finalizeJob calls JobHistory.markCompleted
which breaks stuff badly.
So, I propose a simpler solution: let's move the code in garbageCollect which
does cleanup of:
localJobFile, job-system-dir,
DelegationTokenRenewal.removeDelegationTokenRenewalForJob and fs.close
Then JIP.garbageCollect can use that too.
Thoughts?
> Cleanup required when mapreduce.job.restart.recover is set to false
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-5158
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5158
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 1.2.0
> Reporter: yeshavora
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5158-br1.patch
>
>
> When mapred.jobtracker.restart.recover is set as true and
> mapreduce.job.restart.recover is set to false for a MR job, Job clean up
> never happens for that job if JT restarts while job is running.
> .staging and job-info file for that job remains on HDFS forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
