[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Lu updated YARN-291:
-------------------------
Summary: Dynamic resource configuration (was: Dynamic resource
configuration on NM)
bq. I have concerns on changing the available NMs capacity directly in the NMs.
You probably haven't looked at the patch (YARN-311), which actually changes the
global resource view of RM directly. NMs themselves are untouched. Changed
title to reduce further confusion.
bq. This would complicate significantly allocation changes/re-computation of
resources for running applications.
How so? In fact there is no need to change anything in NM, AFAICT. The existing
containers can either be left alone until completion or suspended/moved/killed
by a management agent.
bq. An (unmanaged) AM requests unmanaged resources from the RM (a new flag in a
resource request)...
First this requires modifying existing applications to support the new request
API, which has significant issues (best-effort, timeout) without preemption.
Second, RM scheduling is fundamentally high latency and best-effort by design
at container granularity. It's not suitable for low latency workload with high
memory utilization via fine grained (page-level) memory sharing. Using the
HBase example again, you'll need to request an "unmanaged resource" with a
container size of at least the peak memory of the region server JVM, which
significantly under utilize memory resource, as peak memory size is
significantly higher than average memory usage.
Note, the existing simple patch doesn't change existing API/protocols. It works
with any existing unmodified OLTP applications and can achieve maximum resource
utilization with mainstream hypervisors.
> Dynamic resource configuration
> ------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
