Wangda Tan commented on YARN-4108:

Uploading design doc with initial POC patch for review.

I put general ideas to design doc, and you can also take a look at POC patch if 
you interested in implementation details. "Background" part could be skipped if 
you have understandings of preemption logic implementation.

What I have done in the POC patch:
- Add a "dryrun" flag to CapacityScheduler to do preemption checks.
- Implemented PreemptionMonitor (major code comes from 
ProportionalCapacityPreemptionPolicy). See ProportionalCapacityMonitorPolicy.
- Implemented PreemptionManager, it stores relationship between demanding apps 
(which comes from under-satisifed queues) and to-be-preempted containers (which 
comes from over-satisifed queues).
And also it can make preemption decision according to ideal allocation 
(calculated by PreemptionMonitor).
- Add a simple unit test to demonstrate how it works from end-to-end. (Allocate 
a 4G container which needs to preempt 4 * 1G container on a same node).

TODO items:
- Reserved container preemption should be also considered
- Optimize how to select to-be-preempt containers, selected containers should 
have minimal impact
- Similar to above, when PreemptionManager found a better combination of 
containers to be preempted (such as containers from lower-priority 
applications), it should be able to update preemption target(s).
- Many other corner cases.

Any review comments / suggestions are welcome! And please feel free to let me 
know if you have any questions. 
+ [~curino], [~jlowe], [~eepayne], [~sunilg]


> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to