Rohit Agarwal commented on YARN-3633:

So, in essence the problem is that when there are too many queues, the fair 
share of each queue gets low and thus the maxAMShare, which is calculated from 
the fairShare of each queue, gets too low to run any container.

I propose the following solution:
Instead of setting
maxAMShare = 0.5*fairShare
we set it to
maxAMShare = max(0.5*fairShare, SomeMinimumSizeEnoughToRunOneContainer)
And then add a cluster-wide maxAMShare to be {{0.5*totalClusterCapacity}}

All these ratios/values can be configurable.

So, in the scenario described in the JIRA, we would still run AMs in some 
queues but we won't overrun the cluster with AMs because it will hit the 
cluster-wide limit.

If this proposal sounds reasonable, I can start working on this.

However, I am not sure how this would interact with preemption.

> With Fair Scheduler, cluster can logjam when there are too many queues
> ----------------------------------------------------------------------
>                 Key: YARN-3633
>                 URL: https://issues.apache.org/jira/browse/YARN-3633
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Rohit Agarwal
>            Assignee: Rohit Agarwal
>            Priority: Critical
> It's possible to logjam a cluster by submitting many applications at once in 
> different queues.
> For example, let's say there is a cluster with 20GB of total memory. Let's 
> say 4 users submit applications at the same time. The fair share of each 
> queue is 5GB. Let's say that maxAMShare is 0.5. So, each queue has at most 
> 2.5GB memory for AMs. If all the users requested AMs of size 3GB - the 
> cluster logjams. Nothing gets scheduled even when 20GB of resources are 
> available.

This message was sent by Atlassian JIRA

Reply via email to