Eric Payne commented on YARN-4781:

[~sunilg], I can think of a couple of ways to handle this, but none of them 
seem ideal to me.

The goal is to sort the apps ordered in the same way the {{FairOrderingPolicy}} 
would sort them, and to do that, we need to use the same logic that is used in 

- One way would be to just copy the compare logic to 
{{IntraQueueCandidatesSelector#TAFairOrderingComparator}}. This is not ideal 
because it introduces a maintenance concern of keeping the two in sync.
- We could split out fifo and fair {{...IntraQueuePreemptionPlugin}}s and the 
new {{FairIntraQueuePreemptionPlugin}} could subclasses 
{{FairOrderingPolicy#FairComparator}}. This seems overly complicated and 
- Add an accessor and all the necessary support to allow the queue's 
orderingpolicy's comparator to be retrieved and used. This would affect fair, 
fifo, and fifo pending policies as well as the interface itself.
- Create a new {{LeafQueue#getAllOrderedApplications}} (to be used instead of 
{{getAllApplications}}) that creates a priority queue sorted according to the 
queue's {{orderingPolicy}}. Or, we could change {{getAllApplications}}| to do 
the same thing. Since a HashSet isn't ordered, nothing that calls 
{{getAllApplications}} today would expect any order.

I like the last option the best. There would still need to be some 
modifications within {{FifoIntraQueuePreemptionPlugin}}, but at least then it's 
all contained within preemption except for the minor mods to {{LeafQueue}}

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Major
>         Attachments: YARN-4781.001.patch, YARN-4781.002.patch, 
> YARN-4781.003.patch
> We introduced fairness queue policy since YARN-3319, which will let large 
> applications make progresses and not starve small applications. However, if a 
> large application takes the queue’s resources, and containers of the large 
> app has long lifespan, small applications could still wait for resources for 
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to 
> preempt resources of queue with fairness policy enabled.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to