Vinod Kumar Vavilapalli commented on YARN-2069:

I had a long look at over the last weekend. This is what the patch does as I 
understand it
 - We are performing two passes - one pass to redistribute based on user-limits 
and one more pass over apps in reverse order of submission
 - Both the passes are best case - we just keep marking containers for 
preemption till we get back enough to satisfy queue capacities

Let's take an example:
 - Let's assume there is a queue with guaranteed capacity of 30%, current 
capacity of 50% and we want it to shrink down to 40% to satisfy some other 
queue's capacities.
 - Assume, 5 different users in this queue and with a minimum user limit 
percentage configured to be 20% of the queue(i.e. no more than 5 users will run 
concurrently in this queue).
 - So current state is that each user has resources of 8%, 8%, 8%, 8%, 8%.

Depending on how users came into the system, the patch may bring it down to 
10%, 10%, 8%, 6%, 6% - it'd still be an unbalanced user share within the queue.

Further, the current user-limit calculation seems to be based on _current 
capacity_ and not _targeted capacity_. So, I am kind of surprised the patch 
preempts containers to satisfy any containers at all.

> CS queue level preemption should respect user-limits
> ----------------------------------------------------
>                 Key: YARN-2069
>                 URL: https://issues.apache.org/jira/browse/YARN-2069
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Mayank Bansal
>         Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch, 
> YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch
> This is different from (even if related to, and likely share code with) 
> YARN-2113.
> YARN-2113 focuses on making sure that even if queue has its guaranteed 
> capacity, it's individual users are treated in-line with their limits 
> irrespective of when they join in.
> This JIRA is about respecting user-limits while preempting containers to 
> balance queue capacities.

This message was sent by Atlassian JIRA

Reply via email to