[
https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570953#comment-14570953
]
David Morel commented on MAPREDUCE-3971:
----------------------------------------
Wondering as well, or wether I should open a separate ticket. What I'm noticing
when accessing /ws/v1/history/mapreduce/jobs/{jobid} for jobs having over 100
Ktasks is a sharp rise in MemHeapUsedM and a server that becomes unresponsive,
even though the heap is set pretty high. I suspect this is due to parsing the
whole jhist file, which I've seen instances of over 900MB. As a result, to
extract stats that I need, I had to write my own external script to parse the
jhist files retrieved from HDFS. Moreover, i noticed some bad interactions when
using oozie (tasks seen as KILLED even though the log says they're SUCCEEDED),
making the history server a SPOF in my architecture...
> Job History web services need to have limits on the number of items they can
> return.
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3971
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Robert Joseph Evans
>
> The Job History web services canput a very large load on the job history
> server. We should put in a limit on the number of entries that can be
> returned by the web service, and also add in the ability to modify the
> starting location in the list, so that all entries can still be downlaoded.
> Just not all at once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
