[
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096987#comment-15096987
]
Karthik Kambatla commented on YARN-1011:
----------------------------------------
bq. Does this mean a queue with SLA sensitive jobs could get OPPORTUNISTIC
containers?
You bring up a good point.
Let us consider an example - an app requests for a container at time 0 that
would run for 30 seconds. Today, let us say, the app would have gotten the
container 10s from submission. With opportunistic scheduling, let us say we are
able to schedule an OPPORTUNISTIC container at 5s. Now, if we have resource
contention within the next 5 seconds, we would just preempt this OPPORTUNISTIC
container and it would get scheduled at 10s as it would have otherwise. If the
contention were to surface at 15s from submission, we would have to preempt and
re-allocate losing 5s compared to what would have happened today. I was hoping
the ability to tune allocation and preemption thresholds would greatly reduce
the likelihood of preemptions, but we can't rely on that for SLA jobs.
I am wary of allowing applications to ask for only GUARANTEED containers, as
that would complicate the scheduler significantly. Since applications are
notified of the ExecutionType, is it okay to leave it upto them to reject
OPPORTUNISTIC containers. We could add this to AMRMClient and MapReduce.
> [Umbrella] Schedule containers based on utilization of currently allocated
> containers
> -------------------------------------------------------------------------------------
>
> Key: YARN-1011
> URL: https://issues.apache.org/jira/browse/YARN-1011
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
> Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf,
> yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated
> containers and, if appropriate, allocate more (speculative?) containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
