[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542297#comment-13542297
]
Luke Lu commented on YARN-291:
------------------------------
A simpler and more straightforward alternative would be changing the node
resource on RM directly via JMX. No protocol changes would be necessary.
Changing resource on NM and propagating the change to RM would lead to nasty
race conditions where RM still thinks that an NM has enough resource and
schedule new containers on the already downsized NM, which should fail and RM
would need to reschedule the containers elsewhere, which is not pretty,
especially when large number of NMs are affected, even if all the corner cases
are handled.
bq. we should do RPC and optionally the web-services to the NM directly. And we
should add command line util too.
Since the only near term user of the feature is *external* admin processes that
prefer not having Hadoop jar dependencies, JMX change alone (a much smaller
patch that doesn't impact major production code paths) would suffice. YARN RPC
changes can be in a separate JIRA, when YARN itself needs to change per node
resources. I'm not sure how useful a YARN CLI interface for this is, given it's
impractical for people to manually change per node resources (when number of
nodes is greater than 10). OTOH, I'm fine with having everything in, as long as
it doesn't delay the inclusion of the basic functionality.
> Dynamic resource configuration on NM
> ------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
