[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064047#comment-14064047
]
Vinod Kumar Vavilapalli commented on YARN-2069:
-----------------------------------------------
I had a long look at over the last weekend. This is what the patch does as I
understand it
- We are performing two passes - one pass to redistribute based on user-limits
and one more pass over apps in reverse order of submission
- Both the passes are best case - we just keep marking containers for
preemption till we get back enough to satisfy queue capacities
Let's take an example:
- Let's assume there is a queue with guaranteed capacity of 30%, current
capacity of 50% and we want it to shrink down to 40% to satisfy some other
queue's capacities.
- Assume, 5 different users in this queue and with a minimum user limit
percentage configured to be 20% of the queue(i.e. no more than 5 users will run
concurrently in this queue).
- So current state is that each user has resources of 8%, 8%, 8%, 8%, 8%.
Depending on how users came into the system, the patch may bring it down to
10%, 10%, 8%, 6%, 6% - it'd still be an unbalanced user share within the queue.
Further, the current user-limit calculation seems to be based on _current
capacity_ and not _targeted capacity_. So, I am kind of surprised the patch
preempts containers to satisfy any containers at all.
> CS queue level preemption should respect user-limits
> ----------------------------------------------------
>
> Key: YARN-2069
> URL: https://issues.apache.org/jira/browse/YARN-2069
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Mayank Bansal
> Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
> YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch
>
>
> This is different from (even if related to, and likely share code with)
> YARN-2113.
> YARN-2113 focuses on making sure that even if queue has its guaranteed
> capacity, it's individual users are treated in-line with their limits
> irrespective of when they join in.
> This JIRA is about respecting user-limits while preempting containers to
> balance queue capacities.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
