[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618697#comment-13618697
]
Luke Lu commented on YARN-291:
------------------------------
bq. The NM is the node controller and responsible for managing the node
resources.
There is actually no/zero code in NM itself (other than picking up resource
config and report to RM) so far that cares about how much resource the node
has. NM is responsible for managing the containers that scheduler assigns to
it. It does not need to know how much resource it really has. Only scheduler
needs to care about how much resource a node has. Making NM oblivious to the
total resource it has obviate the unnecessary split brain situation.
bq. if it had more work to schedule and there were free resources then it would
have already done so
The point was about scheduling latency. There could be a window of OLTP surge
where the nodes have free resource and large job is submitted. OLTP workload is
real time. Also contacting NM to reconfig resource node by node is inefficient
and wasteful.
bq. the currently running tasks will hamper the HBase case even if future work
is not scheduled.
This is orthogonal to scheduling, which this JIRA is about. Management agent
can suspend/move/kill the containers if necessary.
bq. What we need is some way to allow HBase higher priority for system
resources so that tasks do not impede HBase perf. Something like YARN-443.
OS scheduling priority works with CPU and to a degree IO (depending on the IO
scheduler) but not memory, which is critical for many OLTP workload.
bq. This JIRA is part of HADOOP-9165 but that one has been resolved as invalid.
I don't know what you are trying to say. HADOOP-9165 is resolved due to wrong
JIRA component. It could be moved to YARN or MAPREDUCE, but the creator chose
to file new ones instead.
> Dynamic resource configuration on NM
> ------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
