[
https://issues.apache.org/jira/browse/HADOOP-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661533#action_12661533
]
Sharad Agarwal commented on HADOOP-4906:
----------------------------------------
It struck to me that there might be a better approach to the one we have taken,
since the present one is hard to maintain.
Currently there is a new JobConf object created for each task via new
JobConf(rjob.jobFile) in TaskTracker#localizeJob. Since this object is created
from a file, new references are created for all properties. Instead of that
what if we create JobConf as below:
{code}
//keep a reference to JobConf in RunningJob
rjob.localized = true;
rjob.jobConf = localJobConf;
{code}
{code}
//create the task level JobConf
launchTaskForJob(tip, new JobConf(rjob.jobConf));
{code}
This way the properties for task's conf will be shallow cloned resulting in
only reference copy instead of full new object creation. Also there would be a
side benefit as there is no need to parse xml file and create JobConf for each
task.
Thoughts ?
> TaskTracker running out of memory after running several tasks
> -------------------------------------------------------------
>
> Key: HADOOP-4906
> URL: https://issues.apache.org/jira/browse/HADOOP-4906
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Assignee: Sharad Agarwal
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 4906_v1.patch, 4906_v2.patch, 4906_v3.patch
>
>
> Looks like the TaskTracker isn't cleaning up correctly after completed/failed
> tasks, I suspect that the JobConfs aren't being deallocated. Eventually the
> TaskTracker runs out of memory after running several tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
