[
https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261080#comment-17261080
]
VADAGA ANANYO RAO commented on YARN-10559:
------------------------------------------
[~sunilg] I have updated the latest patch and design doc. The FairShare
calculation for preemption in this patch is:
{code:java}
fairSharePerApp = total Queue Cap / no: of apps in the queue
idealFairSharePerAppwithUL = UL / no: of apps of that user
if(apps of user * fairSharePerApp > UL)
fairSharePerApp = idealFairSharePerAppwithUL;
{code}
Say we have 2 users and following scenario1 with UL = 100%:
{noformat}
User1: (UL: 100%)
-App1 (used = 50%, pending = 100%, FS = 33%)
-App3 (used = 50%, pending = 100%, FS = 33%)
User2 (UL: 100%)
-App3 (used = 0%, pending = 100%, FS = 33%)
{noformat}
We will take 17% resources from app1 and app2 to give to app3.
Now, say we have 2 users with UL = 50%:
{noformat}
User1: (UL: 50%)
-App1 (used = 50%, pending = 100%, FS = 25%)
-App3 (used = 50%, pending = 100%, FS = 25%)
User2 (UL: 50%)
-App3 (used = 0%, pending = 100%, FS = 33%)
{noformat}
We will take ~17% resources from app1 and app2 to give to app3. We may be
under-preempting from user1 in this case. But we ensure user2 is not heavily
starved.
This logic seems to be functioning closest to the FairOrderingPolicy in
CapacityScheduler.
Comparisons of other considered algorithms is present in design_doc_v2.
Once again, thank you [~sunilg] for guiding with the design.
[~epayne] [~leftnoteasy] Please do review the latest patch-2.
Thank you.
> Fair sharing intra-queue preemption support in Capacity Scheduler
> -----------------------------------------------------------------
>
> Key: YARN-10559
> URL: https://issues.apache.org/jira/browse/YARN-10559
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Affects Versions: 3.1.4
> Reporter: VADAGA ANANYO RAO
> Assignee: VADAGA ANANYO RAO
> Priority: Major
> Fix For: 3.1.4
>
> Attachments: FairOP_preemption-design_doc_v1.pdf,
> FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch,
> YARN-10559.0002.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Usecase:
> Due to the way Capacity Scheduler preemption works, If a single user submits
> a large application to a queue (using 100% of resources), that job will not
> be preempted by future applications from the same user within the same queue.
> This implies that the later applications will be forced to wait for
> completion of the long running application. This prevents multiple long
> running, large, applications from running concurrently.
> Support fair sharing among apps while preempting applications from same queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
