[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075220#comment-14075220
]
Wangda Tan commented on YARN-2069:
----------------------------------
Hi Mayank,
Thanks for your detailed explanation, I think I understood your approach.
However, I think the current way to compute target user limit is not correct,
let me explain:
I found basically, your created {{computeTargetedUserLimit}} is modified from
{{computeUserLimit}}, it will calculate as following
{code}
target_capacity = used_capacity - resToObtain
min(
max(target_capacity / #active_user,
target_capacity * user_limit_percent),
target_capacity * user_limit_factor)),
{code}
So when a user_limit_percent is set as default (100%), it is possible that
target_user_limit * #active_user > queue_max_capacity.
In this case, it is possible that any of the user-usage is below
target_user_limit, but the usage of the queue is larger than guaranteed
resource.
Let me give you an example
{code}
Assume queue capacity = 50, used_resource = 70, resToObtain = 20
So target_capacity = 50, there're 5 users in the queue
user_limit_percent = 100%, user_limit_factor = 1 (both are default)
So target_user_capacity = min(max(50 / 5, 50 * 100%), 50) = 50
User1 used 20
User2 used 10
User3 used 10
User4 used 20
User5 used 10
So all user's used capacity are < target_user_capacity
{code}
In existing logic of {{balanceUserLimitsinQueueForPreemption}}
{code}
if (Resources.lessThan(rc, clusterResource, userLimitforQueue,
userConsumedResource)) {
// do preemption
} else
continue;
{code}
If a user used resource < target_user_capacity, it will not be preempted.
Mayank, is that correct? Or I misunderstood your logic? Please let me know you
comments,
Thanks,
Wangda
> CS queue level preemption should respect user-limits
> ----------------------------------------------------
>
> Key: YARN-2069
> URL: https://issues.apache.org/jira/browse/YARN-2069
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Mayank Bansal
> Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
> YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch,
> YARN-2069-trunk-6.patch, YARN-2069-trunk-7.patch
>
>
> This is different from (even if related to, and likely share code with)
> YARN-2113.
> YARN-2113 focuses on making sure that even if queue has its guaranteed
> capacity, it's individual users are treated in-line with their limits
> irrespective of when they join in.
> This JIRA is about respecting user-limits while preempting containers to
> balance queue capacities.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
