[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330649#comment-15330649
]
Inigo Goiri commented on YARN-5215:
-----------------------------------
[~kasha], in our use case we are targeting co-locating with latency sensitive
workloads and they have diurnal patterns. For this type of workload, we need to
be fairly reactive. Actually, preempting containers at the NM following the
{{ContainersMonitor}} loop would be ideal.
The improvements in utilization are significant as right now we are just
reserving for the peak of the latency sensitive workloads (around ~50%) of the
machine. We tried at some point to have a separate service to periodically
change the resources of the NMs but it's harder to operate.
In any case, in this first patch, we are just preventing scheduling containers
and not adding preemption. I can add the following changes to the current patch:
# UI improvements
# History in the utilization to take decisions
# Preempting containers from the RM
# Preempting containers from the NM
The problem with preemption is that we would go into what to preempt and that
might have some dependencies in the opportunistic stuff in YARN-1011.
> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch, YARN-5215.001.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is consumed by external processes and
> schedule based on this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
