Robert Kanter created MAPREDUCE-6970: ----------------------------------------
Summary: archive-logs tool should throttle container requests Key: MAPREDUCE-6970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0-alpha1, 2.8.0 Reporter: Robert Kanter The {{mapred archive-logs}} command currently has no way to throttle the number of requested containers. For example, we recently saw a busy cluster where the tool hadn't been run for a while and there were about 20,000 apps to process. This meant that the tool tried to request 20,000 containers and got a ton of GC and then OOM trying to handle that. This problem can be mitigated by setting {{-maxEligibleApps}} to a more reasonable value, but doing so would require running the tool multiple times; plus, the default value is {{-1}} (all). We should add a way to throttle the max number of concurrently running containers that the tool manages. Something like {{-concurrency <n>}} where it would only allow up to {{n}} containers at a time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org