Wangda Tan commented on YARN-4108:

Thanks [~jlowe]/[~sunilg] for sharing your thoughts!

I think the ideal case is as what [~jlowe] mentioned: {{When we decide to 
preempt we can move the request's reservation to the node where we decided to 
preempt containers.}}. If ProportionalCPP has this capability, most of the 
problems will be resolved. But one major issue is, it will be hard to handle 
requests and preemption together -- We have a very complex ordering algorithm 
to determine which request/application will be served, we will do some 
simulation to get the ordering, which is very expensive.

Along with Jason's suggestion, we can achieve this by implementing algorithm 
within scheduler's allocation cycle, which can simplify preemption logic AND 
makes preemption logic synchronized with allocation logic.
- ProportionalCPP will decide how to balance resources between queues, which is 
as simple as: how much of resources of each queue/application/user can be 
preempted. Please note that, PCPP won't mark any of to-be-preempted container 
in this stage.
- Scheduler's allocation cycle will proactively trigger preemption, we don't 
need to do this every node heartbeat. To make this efficient, *we can do the 
preemption check once every X (configurable) node heartbeat.*

Logic may look like:

node-do-heartbeat(node, application) {
        if (do-preemption-check) {
                // Preemptable resource is decided by ProportionalCPP AND the 
                Resource preemptable = node.getPreemptable(application)

                if (node.available + preemptable > application.next_request) {
                        // mark to-be-preempted containers
                } else {
                        // reserve application.next_request if it could be 
                        // preemptable containers will keep running if reserved 
container can be allocated

This has same order of magnitudes time complex, and it should be able to solve 
the progressive preemption problem.

cc: [~curino].

> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to