[
https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289680#comment-13289680
]
Herman Chen commented on MAPREDUCE-4304:
----------------------------------------
I tried CapacityScheduler but I'm still running into the problem. My setup
this time: there are 4 NM each with 1.5GB container memory configured. I'm
using the default value of 0.1 (10%) for
yarn.scheduler.capacity.maximum-am-resource-percent, and that ends up giving me
a "Max Active Applications" of 5 for the default queue (initially I thought 10%
would give me just 1 Application). And again when I submit 11 jobs
simultaneously, one AM is allocated on each of the NM, taking up all resources
(default AM size is 1.5GB). To make this work for my setup, I have to
configure the property to at most 0.06, which gives me "Max Active
Applications" of 3. So it looks like yes it can be done, but definitely not
intuitively.
The discrepancy lies in how YARN plans resource usage versus how submitted jobs
actually allocate resource. The way the maximum-am-resource-percent is used is
total memory (6G) / minimum allocation (128MB) * max-am-percent (0.1) =~ 5.
But this uses minimum allocation, does that make sense? For my jobs with
default AM size of 1.5G, it certainly cannot support 5 applications.
It feels to me, the max-am-percent is more intuitively enforced in terms of
memory, instead of translating it into number of applications. Alternatively a
mechanism to preempt idle AMs that cannot allocate further resources seems
valuable.
> Deadlock where all containers are held by ApplicationMasters should be
> prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container
> memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is
> likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but
> then the jobs block each other indefinitely because they're all waiting to
> allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as
> the number of jobs being submitted is greater than the rate jobs finish, it
> may run into a vicious cycle where more and more containers are locked up by
> ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
