[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053959#comment-14053959
]
Mayank Bansal commented on YARN-2069:
-------------------------------------
hi [~wangda],
Thanks for the review. I updated the patch please take a look , Le tme answer
your questions.
bq. In ProportionalCapacityPreemptionPolicy,
bq. 1) balanceUserLimitsinQueueForPreemption()
bq. 1.1, I think there's a bug when multiple applications under a same user
(say Jim) in a queue, and usage of Jim is over user-limit.
Any of Jim's applications will be tried to be preempted
(total-resource-used-by-Jim - user-limit).
We should remember resourcesToClaimBackFromUser and initialRes for each user
(not reset them when handling each application)
And it's better to add test to make sure this behavior is correct.
We need to maintian the reverse order of application submission which only can
be done iterating through applications as we want to preempt applications which
are last submitted.
bq. 1.2, Some debug logging should be removed like
Done
bq. 1.3, This check should be unnecessary
Done
bq. 2) preemptFrom
bq. I noticed this method will be called multiple times for a same application
within a editSchedule() call.
bq. The reservedContainers will be calculated multiple times.
bq. An alternative way to do this is to cache
This method will only be executed for all the applicatoins only once as we will
be removing all reservations and for the apps the reservation is been removed
that would be no-op
bq.In LeafQueue,
bq. 1) I think it's better to remember user limit, no need to compute it every
time, add a method like getUserLimit() to leafQueue should be better.
That valus is not static and changed every time based on cluster utilization
and thats why i am calculating every time.
bq, 1) Should we preempt containers equally from users when there're multiple
users beyond user-limit in a queue?
Its not good it should be based on who submitted last and over user limit,
however its not fair but we want to preempt last jobs first
bq. 2) Should we preempt containers equally from applications in a same user?
(Heap-like data structure maybe helpful to solve 1/2)
No as above mentioned
bq. 3) Should user-limit preemption be configurable?
I think if we just configure preemption , thats enough thoughts?
Thanks,
Mayank
> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>
> Key: YARN-2069
> URL: https://issues.apache.org/jira/browse/YARN-2069
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Mayank Bansal
> Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
> YARN-2069-trunk-3.patch
>
>
> Preemption today only works across queues and moves around resources across
> queues per demand and usage. We should also have user-level preemption within
> a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
