[
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383904#comment-16383904
]
Eric Payne commented on YARN-4781:
----------------------------------
My thinking on a solution to this would be to check in
{{FifoIntraQueuePreemptionPlugin}} whether or not the FairOrderingPolicy is
enabled on the queue being processed. In {{createTempAppForResCalculation}} and
{{calculateToBePreemptedResourcePerApp}}, if the FairOrderingPolicy is enabled
for the queue, use a different comparator (other than {{TAPriorityComparator}})
that has the following algorithm:
I have a patch coded up if you would be interested.
{code:title=TAFairOrderingComparator}
if ((total resources used by user of app1) == (total resources used by user
of app2)) {
return appID1 - appID2
}
if ((total resources used by user of app1) < (total resources used by user
of app2)) {
return -1
} else {
return 1
}
{code}
I have a patch coded up if you would be interested.
> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>
> Key: YARN-4781
> URL: https://issues.apache.org/jira/browse/YARN-4781
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Major
>
> We introduced fairness queue policy since YARN-3319, which will let large
> applications make progresses and not starve small applications. However, if a
> large application takes the queue’s resources, and containers of the large
> app has long lifespan, small applications could still wait for resources for
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to
> preempt resources of queue with fairness policy enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
