[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618522#comment-13618522
]
Luke Lu commented on YARN-291:
------------------------------
Resource scheduling is fundamentally centralized at RM. The global resource
view is currently bootstrapped via the node registration process, which is more
of a historical artifact based on convenience, since the resource view can also
be constructed directly on RM via an inventory database. It's a round about,
inconvenient and inefficient way to (re)construct the resource view by
modifying per node config explicitly and propagate partial views to RM, if you
already have an inventory database.
For a practical example: if you have OLTP workload (say HBase) sharing the same
hardware with YARN and there is a load surge on HBase, we need to stop
scheduling tasks/containers immediately on relevant (potentially all) nodes.
The current patch (JMX is just used as a portable protocol for external
management client to communicate with RM) can take effect immediately most
efficiently. If we explicitly modify each nodemanager config and let the NM-RM
protocol to propagate the change to RM, it would waste resource (CPU and
network bandwidth) to contact (potentially) all the nodemanagers and cause
unnecessary scheduling delays if the propagation is via regular heartbeat
and/or DDoS the RM if (potentially) all the NMs need to re-register out-of-band.
This is not about JMX based tricks. This is about changing global resource view
directly where the scheduler is vs the Rube Goldbergish way of changing NM
config individually and propagate changes to RM to reconstruct the resource
view. IMO, the direct way is better because NM doesn't really care about what
resource it really has.
> Dynamic resource configuration on NM
> ------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
