Robert Kanter created MAPREDUCE-6970:
----------------------------------------
Summary: archive-logs tool should throttle container requests
Key: MAPREDUCE-6970
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0-alpha1, 2.8.0
Reporter: Robert Kanter
The {{mapred archive-logs}} command currently has no way to throttle the number
of requested containers. For example, we recently saw a busy cluster where the
tool hadn't been run for a while and there were about 20,000 apps to process.
This meant that the tool tried to request 20,000 containers and got a ton of GC
and then OOM trying to handle that.
This problem can be mitigated by setting {{-maxEligibleApps}} to a more
reasonable value, but doing so would require running the tool multiple times;
plus, the default value is {{-1}} (all).
We should add a way to throttle the max number of concurrently running
containers that the tool manages. Something like {{-concurrency <n>}} where it
would only allow up to {{n}} containers at a time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
