Wangda Tan commented on YARN-2069:

Hi Mayank,
Thanks for you patch, I just looked at it, some comments on implementation,

*In ProportionalCapacityPreemptionPolicy,*
1) balanceUserLimitsinQueueForPreemption()
1.1, I think there's a bug when multiple applications under a same user (say 
Jim) in a queue, and usage of Jim is over user-limit.
Any of Jim's applications will be tried to be preempted 
(total-resource-used-by-Jim - user-limit).
We should remember resourcesToClaimBackFromUser and initialRes for each user 
(not reset them when handling each application)
And it's better to add test to make sure this behavior is correct.

1.2, Some debug logging should be removed like
LOG.info("MAYANK queue NAME "+ qT.leafQueue.getQueueName());

1.3, This check should be unnecessary
      if(userLimitforQueue == null){

2) preemptFrom
I noticed this method will be called multiple times for a same application 
within a editSchedule() call.
The reservedContainers will be calculated multiple times.
An alternative way to do this is to cache
    List<RMContainer> containers =
      new ArrayList<RMContainer>(app.getLiveContainers());
At the begining of getContainersToPreempt(), clone them to a set for these 
running containers. And remove container from the set if we processed it.
Another benefit to do is eliminate checking like,
        Set<RMContainer> userContainers = userLimitContainers.get(app
        if (userContainers.contains(c)) {
And preemptFrom called by balanceUserLimitsinQueueForPreemption passed dumb 
skippedAMContainerlist/skippedAMSize in. It's better to create a overloaded 
preemptFrom to make code can be easier read.

*In LeafQueue,*
1) I think it's better to remember user limit, no need to compute it every 
time, add a method like getUserLimit() to leafQueue should be better.

*In beyond, questions about some undefined requirements,*
1) Should we preempt containers equally from users when there're multiple users 
beyond user-limit in a queue?
2) Should we preempt containers equally from applications in a same user? 
(Heap-like data structure maybe helpful to solve 1/2)
3) Should user-limit preemption be configurable?

The last is as [~sunilg] suggested,  your patch is based on YARN-2022, it's 
better to separate your patch from YARN-2022 for easier review. You can point 
which patch in YARN-2022 you based on. 


> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>                 Key: YARN-2069
>                 URL: https://issues.apache.org/jira/browse/YARN-2069
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Mayank Bansal
>         Attachments: YARN-2069-trunk-1.patch
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.

This message was sent by Atlassian JIRA

Reply via email to