[
https://issues.apache.org/jira/browse/YARN-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266673#comment-15266673
]
Jason Lowe commented on YARN-4280:
----------------------------------
bq. The problem of allowing one container reserved exceed queue's max capacity
is: in a small cluster, one single large container could mean a large
proportion of the cluster. And queue's maximum capacity will be exceeded a lot.
Agreed, we don't want to let a thousand users in a relatively small queue lock
down the entire cluster with reservations.
I think the algorithm needs to work something like this:
- If a user would normally need to make a reservation and is within its user
limits but the queue absolute capacity would be exceeded then that queue is
locked down for further container allocations until any of the following are
true:
-- the container can be allocated
-- the reservation can be placed
-- the user limit is lowered and the reservation would no longer fit within the
user limit
-- the container request is cancelled
-- a higher priority allocation comes along that can fit
- If a user would normally need to make a reservation and is within both its
user limits and leaf queue absolute capacity _but_ a parent queue's capacity
would be exceeded then _all_ queues within that parent queue's hierarchy are
locked down from further allocations until any of the following are true:
-- the container can be allocated
-- the reservation can be placed
-- the user limit is lowered and the reservation would no longer fit within the
user limit
-- the container request is cancelled
-- a higher priority allocation (e.g.: from a more underserved queue within the
hierarchy) comes along that can fit
The leaf queue case seems pretty straightforward -- we simply early-out of the
assignment loop if we need to make a reservation but cannot due to the queue's
limits. Then only higher-priority allocations will be allowed to be made until
we can place that reservation.
The parent queue case is more complicated, but I think it can be accomplished
in a similar manner. When the parent-queue limit is detected by the leaf queue
assignment method, it returns with a result that indicates to the parent
queue(s) that no further allocations can be made due to parent queue limits,
and all queues up to the parent queue that also hit that limit return early
from the assignment due to that condition. Then only higher-priority
allocations (i.e.: from either a more underserved queue in the hierarchy or
higher-priority app within the leaf queue) could potentially get more
containers until the pertinent container request can be addressed either by
reservation or allocation.
> CapacityScheduler reservations may not prevent indefinite postponement on a
> busy cluster
> ----------------------------------------------------------------------------------------
>
> Key: YARN-4280
> URL: https://issues.apache.org/jira/browse/YARN-4280
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Affects Versions: 2.6.1, 2.8.0, 2.7.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
>
> Consider the following scenario:
> There are 2 queues A(25% of the total capacity) and B(75%), both can run at
> total cluster capacity. There are 2 applications, appX that runs on Queue A,
> always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2
> GB containers.
> The user limit is high enough for the application to reach 100% of the
> cluster resource.
> appX is running at total cluster capacity, full with 1G containers releasing
> only one container at a time. appY comes in with a request of 2GB container
> but only 1 GB is free. Ideally, since appY is in the underserved queue, it
> has higher priority and should reserve for its 2 GB request. Since this
> request puts the alloc+reserve above total capacity of the cluster,
> reservation is not made. appX comes in with a 1GB request and since 1GB is
> still available, the request is allocated.
> This can continue indefinitely causing priority inversion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
