Wangda Tan commented on YARN-2069:

Hi Mayank,
Thanks for your detailed explanation, I think I understood your approach.

However, I think the current way to compute target user limit is not correct, 
let me explain:
I found basically, your created {{computeTargetedUserLimit}} is modified from 
{{computeUserLimit}}, it will calculate as following
target_capacity = used_capacity - resToObtain
    max(target_capacity / #active_user,
         target_capacity * user_limit_percent),
    target_capacity * user_limit_factor)),
So when a user_limit_percent is set as default (100%), it is possible that 
target_user_limit * #active_user > queue_max_capacity.
In this case, it is possible that any of the user-usage is below 
target_user_limit, but the usage of the queue is larger than guaranteed 

Let me give you an example
Assume queue capacity = 50, used_resource = 70, resToObtain = 20
So target_capacity = 50, there're 5 users in the queue
user_limit_percent = 100%, user_limit_factor = 1 (both are default)

So target_user_capacity = min(max(50 / 5, 50 * 100%), 50) = 50
User1 used 20
User2 used 10
User3 used 10
User4 used 20
User5 used 10

So all user's used capacity are < target_user_capacity

In existing logic of {{balanceUserLimitsinQueueForPreemption}}
      if (Resources.lessThan(rc, clusterResource, userLimitforQueue,
          userConsumedResource)) {
         // do preemption
      } else 
If a user used resource < target_user_capacity, it will not be preempted.

Mayank, is that correct? Or I misunderstood your logic? Please let me know you 


> CS queue level preemption should respect user-limits
> ----------------------------------------------------
>                 Key: YARN-2069
>                 URL: https://issues.apache.org/jira/browse/YARN-2069
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Mayank Bansal
>         Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch, 
> YARN-2069-trunk-3.patch, YARN-2069-trunk-4.patch, YARN-2069-trunk-5.patch, 
> YARN-2069-trunk-6.patch, YARN-2069-trunk-7.patch
> This is different from (even if related to, and likely share code with) 
> YARN-2113.
> YARN-2113 focuses on making sure that even if queue has its guaranteed 
> capacity, it's individual users are treated in-line with their limits 
> irrespective of when they join in.
> This JIRA is about respecting user-limits while preempting containers to 
> balance queue capacities.

This message was sent by Atlassian JIRA

Reply via email to