[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573204#comment-14573204 ] Ray Chiang commented on MAPREDUCE-3971: --- For .jhist parsing, I've added MAPREDUCE-6376. Job History web services need to have limits on the number of items they can return. Key: MAPREDUCE-3971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans The Job History web services canput a very large load on the job history server. We should put in a limit on the number of entries that can be returned by the web service, and also add in the ability to modify the starting location in the list, so that all entries can still be downlaoded. Just not all at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570953#comment-14570953
]
David Morel commented on MAPREDUCE-3971:
Wondering as well, or wether I should open a separate ticket. What I'm noticing
when accessing /ws/v1/history/mapreduce/jobs/{jobid} for jobs having over 100
Ktasks is a sharp rise in MemHeapUsedM and a server that becomes unresponsive,
even though the heap is set pretty high. I suspect this is due to parsing the
whole jhist file, which I've seen instances of over 900MB. As a result, to
extract stats that I need, I had to write my own external script to parse the
jhist files retrieved from HDFS. Moreover, i noticed some bad interactions when
using oozie (tasks seen as KILLED even though the log says they're SUCCEEDED),
making the history server a SPOF in my architecture...
Job History web services need to have limits on the number of items they can
return.
Key: MAPREDUCE-3971
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.2
Reporter: Robert Joseph Evans
The Job History web services canput a very large load on the job history
server. We should put in a limit on the number of entries that can be
returned by the web service, and also add in the ability to modify the
starting location in the list, so that all entries can still be downlaoded.
Just not all at once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382921#comment-14382921 ] Ray Chiang commented on MAPREDUCE-3971: --- I just ran across this JIRA. I know this one is quite old, but I thought I'd see if anyone here thinks this has overlap with MAPREDUCE-6222. Job History web services need to have limits on the number of items they can return. Key: MAPREDUCE-3971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans The Job History web services canput a very large load on the job history server. We should put in a limit on the number of entries that can be returned by the web service, and also add in the ability to modify the starting location in the list, so that all entries can still be downlaoded. Just not all at once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
