[ 
https://issues.apache.org/jira/browse/YARN-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Shao updated YARN-3807:
---------------------------
    Description: 
This proposal talks about limitations of the YARN scheduling policies for SLA 
applications, and tries to solve them by YARN-3806 and the new scheduling 
policy called guaranteed capacity scheduling.
Guaranteed capacity scheduling makes guarantee to the applications that they 
can get resources under specified capacity cap in totally predictable manner. 
The application can meet SLA more easily since it is self-contained in the 
shared cluster - external uncertainties are eliminated.
For example, suppose queue A has initial capacity 100G memory, and there are 
two pending applications 1 and 2, 1’s specified capacity is 70G, 2’s specified 
capacity is 50G. Queue A may accept application 1 to run first and makes 
guarantee that 1 can get resources exponentially up to its capacity and won’t 
be preempted (if allocation of 1 is 5G in scheduling cycle N, demand is 80G, 
exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G, in N+3, 
it can get 20G, and in N+4, it can get 30G, reach its capacity). Later, when 
the cluster is free, queue A may decide to scale up by increasing its capacity 
to 120G, so it can accept application 2 and make guarantee to it as well. Queue 
A can scale down to its initial capacity when any application completes.
Guaranteed capacity scheduling also has other features that the example doesn’t 
illustrate. See proposal for more details.

  was:
This proposal talks about limitations of the YARN scheduling policies for SLA 
applications, and tries to solve them by YARN-3806 and the new scheduling 
policy called guaranteed capacity scheduling.
Guaranteed capacity scheduling makes guarantee to the applications that they 
can get resources under specified capacity cap in totally predictable manner. 
The application can meet SLA more easily since it is self-contained in the 
shared cluster - external uncertainties are eliminated.
For example, suppose queue A has initial capacity 100G memory, and there are 
two pending applications 1 and 2, 1’s specified capacity is 70G, 2’s specified 
capacity is 50G. Queue A may accept application 1 to run first and makes 
guarantee that 1 can get resources exponentially up to its capacity and won’t 
be preempted (if allocation of 1 is 5G in scheduling cycle N, demand is 80G, 
exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G, in N+3, 
it can get 20G, and in N+4, it can get 30G, reach its capacity). Later, when 
the cluster is free, queue A may decide to scale up by increasing its capacity 
to 120G, so it can accept application 2 and make guarantee to it as well. Queue 
A can scale down to its initial capacity when any application completes.
Guaranteed capacity scheduling also have some other features. See proposal for 
more details.


> Proposal of Guaranteed Capacity Scheduling for YARN
> ---------------------------------------------------
>
>                 Key: YARN-3807
>                 URL: https://issues.apache.org/jira/browse/YARN-3807
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler, fairscheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGuaranteedCapacitySchedulingForYARN-V1.0.pdf
>
>
> This proposal talks about limitations of the YARN scheduling policies for SLA 
> applications, and tries to solve them by YARN-3806 and the new scheduling 
> policy called guaranteed capacity scheduling.
> Guaranteed capacity scheduling makes guarantee to the applications that they 
> can get resources under specified capacity cap in totally predictable manner. 
> The application can meet SLA more easily since it is self-contained in the 
> shared cluster - external uncertainties are eliminated.
> For example, suppose queue A has initial capacity 100G memory, and there are 
> two pending applications 1 and 2, 1’s specified capacity is 70G, 2’s 
> specified capacity is 50G. Queue A may accept application 1 to run first and 
> makes guarantee that 1 can get resources exponentially up to its capacity and 
> won’t be preempted (if allocation of 1 is 5G in scheduling cycle N, demand is 
> 80G, exponential factor is 2. In N+1, it can get 5G, in N+2, it can get 10G, 
> in N+3, it can get 20G, and in N+4, it can get 30G, reach its capacity). 
> Later, when the cluster is free, queue A may decide to scale up by increasing 
> its capacity to 120G, so it can accept application 2 and make guarantee to it 
> as well. Queue A can scale down to its initial capacity when any application 
> completes.
> Guaranteed capacity scheduling also has other features that the example 
> doesn’t illustrate. See proposal for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to