[
https://issues.apache.org/jira/browse/MAPREDUCE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6970:
-------------------------------------
Issue Type: Improvement (was: Bug)
> archive-logs tool should throttle container requests
> ----------------------------------------------------
>
> Key: MAPREDUCE-6970
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.8.0, 3.0.0-alpha1
> Reporter: Robert Kanter
>
> The {{mapred archive-logs}} command currently has no way to throttle the
> number of requested containers. For example, we recently saw a busy cluster
> where the tool hadn't been run for a while and there were about 20,000 apps
> to process. This meant that the tool tried to request 20,000 containers and
> got a ton of GC and then OOM trying to handle that.
> This problem can be mitigated by setting {{-maxEligibleApps}} to a more
> reasonable value, but doing so would require running the tool multiple times;
> plus, the default value is {{-1}} (all).
> We should add a way to throttle the max number of concurrently running
> containers that the tool manages. Something like {{-concurrency <n>}} where
> it would only allow up to {{n}} containers at a time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
