Ray Chiang created MAPREDUCE-6622:
-------------------------------------
Summary: Add capability to set JHS job cache to a task-based limit
Key: MAPREDUCE-6622
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 2.7.2
Reporter: Ray Chiang
Assignee: Ray Chiang
When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs
can be of varying size. This is generally not a problem when the jobs sizes
are uniform or small, but when the job sizes can be very large (say greater
than 250k tasks), then the JHS heap size can grow tremendously.
In cases, where multiple jobs are very large, then the JHS can lock up and
spend all its time in GC. However, since the cache is holding on to all the
jobs, not much heap space can be freed up.
By setting a property that sets a cap on the number of tasks allowed in the
cache and since the total number of tasks loaded is directly proportional to
the amount of heap used, this should help prevent the JHS from locking up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
