Mayank Bansal commented on YARN-2069:

hi [~wangda],

Thanks for the review. I updated the patch please take a look , Le tme answer 
your questions.
bq. In ProportionalCapacityPreemptionPolicy,
bq. 1) balanceUserLimitsinQueueForPreemption()
bq. 1.1, I think there's a bug when multiple applications under a same user 
(say Jim) in a queue, and usage of Jim is over user-limit.
Any of Jim's applications will be tried to be preempted 
(total-resource-used-by-Jim - user-limit).
We should remember resourcesToClaimBackFromUser and initialRes for each user 
(not reset them when handling each application)
And it's better to add test to make sure this behavior is correct.

We need to maintian the reverse order of application submission which only can 
be done iterating through applications as we want to preempt applications which 
are last submitted. 

bq. 1.2, Some debug logging should be removed like

bq. 1.3, This check should be unnecessary

bq. 2) preemptFrom
bq. I noticed this method will be called multiple times for a same application 
within a editSchedule() call.
bq. The reservedContainers will be calculated multiple times.
bq. An alternative way to do this is to cache
This method will only be executed for all the applicatoins only once as we will 
be removing all reservations and for the apps the reservation is been removed 
that would be no-op

bq.In LeafQueue,
bq. 1) I think it's better to remember user limit, no need to compute it every 
time, add a method like getUserLimit() to leafQueue should be better.
That valus is not static and changed every time based on cluster utilization 
and thats why i am calculating every time.

bq, 1) Should we preempt containers equally from users when there're multiple 
users beyond user-limit in a queue?
Its not good it should be based on who submitted last and over user limit, 
however its not fair but we want to preempt last jobs first

bq. 2) Should we preempt containers equally from applications in a same user? 
(Heap-like data structure maybe helpful to solve 1/2)
No as above mentioned

bq. 3) Should user-limit preemption be configurable?
I think if we just configure preemption , thats enough thoughts?


> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>                 Key: YARN-2069
>                 URL: https://issues.apache.org/jira/browse/YARN-2069
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Mayank Bansal
>         Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch, 
> YARN-2069-trunk-3.patch
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.

This message was sent by Atlassian JIRA

Reply via email to