[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13682124#comment-13682124
]
Junping Du commented on YARN-291:
---------------------------------
Hi [~tucu00], thanks for comments above. Actually, just like Luke's comments,
in this JIRA, we tried to address the case that YARN node's resource is changed
by plan as mixing resources with non-yarn applications (like HBase, Drill,
etc.) but not a react to app's short-term thrashing behavior. Thus, we don't
have to change anything related to app's API, but just provide a way to change
node's resource through Admin APIs (Admin Protocol, CLI, REST and JMX). The
only special case is: we should deal with case that running Containers'
resource larger than NM's total resource (after changed), but it can be handled
well by involving a minus value for available resource (we can call it a "debt"
resource model) which shows great flexibility of YARN framework. And YARN
scheduler just stop to assign containers on over-loaded NM (in "debt") until
its resource being balanced again which looks perfect in our previous test (100
virtual nodes on 10 physical servers). In the long term, preemption mechanism
can also be involved for releasing containers/resource in some cases. Thoughts?
> Dynamic resource configuration
> ------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> More background and design details can be found in attached proposal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
