[ http://issues.apache.org/jira/browse/HADOOP-815?page=all ]
Arun C Murthy updated HADOOP-815:
---------------------------------
Attachment: HADOOP-815_20061230_4.patch
jt_memory_profiles.tgz
Ok, here is a reasonably well-tested patch...
List of changes:
a) Fixed HADOOP-740 i.e. ensure task entries are cleaned up on completion.
b) Fixed HADOOP-787 i.e. ensure we keep only 100jobs per user; rest are
available via jobhistroy anyway.
c) Fixed both JobTracker & TaskTracker to ensure lost
status-updates/heartbeatResponses due to lost rpcs are resent by both
TaskTracker & JobTracker; and also that the JobTracker can detect that
duplicate 'TaskTrackerStatus' updates and ignore them, which otherwise are
fatal.
d) Some miscellaneous fixes like using ArrayList instead of TreeSet and array
for 'usableTaskIds' in TaskInProgress.java
Results:
Currently after running smallJobsBenchmark with 750 jobs each with 300 maps &
2 reduces (i.e. total of ~225,000 tasks) the memory footprint of the JobTracker
is ~1.5Gb after 'RETIRE_JOB_INTERVAL' (which I suspect also leads to
degeneration of JT's performance as in HADOOP-843 since each of the JT's
datastructures are extremely bloated leading to sluggishness). With this patch
the memory-footprint is down to ~150MB after 'RETIRE_JOB_INTERVAL', yes, that's
150Mb! :) (and seems to solve HADOOP-843 too).
Appreciate any feedback...
> Investigate and fix the extremely large memory-footprint of JobTracker
> ----------------------------------------------------------------------
>
> Key: HADOOP-815
> URL: http://issues.apache.org/jira/browse/HADOOP-815
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.9.1
> Reporter: Arun C Murthy
> Assigned To: Arun C Murthy
> Fix For: 0.10.0
>
> Attachments: 150k_1199_774.nps, 75k_jobs.nps,
> HADOOP-815_20061220_1.patch, HADOOP-815_20061221_2.patch,
> HADOOP-815_20061222_3.patch, HADOOP-815_20061230_4.patch,
> jt_memory_profiles.tgz
>
>
> The JobTracker's memory footprint seems excessively large, especially when
> many jobs are submitted.
> Here is the 'top' output of a JobTracker which has scheduled ~1k jobs thus
> far:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 31877 arunc 19 0 2362m 261m 13m S 14.0 12.9 24:48.08 java
> Clearly VIRTual memory of 2364Mb v/s 261Mb of RESident memory is symptomatic
> of this issue...
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
