[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618681#comment-13618681
]
Bikas Saha commented on YARN-291:
---------------------------------
bq. because NM doesn't really care about what resource it really has
I dont quite agree with this. The NM is the node controller and responsible for
managing the node resources.
RM-NM information exchange is a core concept in YARN. I dont think I agree with
updating the RM directly and not bothering about what the NM's think or report
about their resources. My concern is about NM and RM going out of sync and YARN
becoming harder to understand.
bq. Since the only near term user of the feature is external admin processes
that prefer not having Hadoop jar dependencies
It would help if some concrete scenarios were described. From the document, I
get the general notion of cloud services having elastic nodes. If the cloud
actually changes the node resources then the NM's should get to know that
something has changed and report it. The reactive race condition is always
there irrespective of whether NM's report the change or some external entity
does. NM's sync with the RM every 1 second. Is that duration too short for the
use cases being targeted? The argument in favor of directly changing the RM
view makes the case for almost all nodes changing state at once in the cluster.
Is that the primary scenario?
The HBase scenario mentioned in the comments needs more thought. Simply
updating the RM with lower NM resource info may not be enough. True, the
updated RM will not schedule more work on them. But the point is that if it had
more work to schedule and there were free resources then it would have already
done so. If resources are free then it means RM does not have work for them. If
resources are not free then RM will not schedule more work until the NM
heartbeat tells the RM about freshly freed up resources. So I dont see how
updating the RM out of band via admin will help.
Also, the currently running tasks will hamper the HBase case even if future
work is not scheduled. What we need is some way to allow HBase higher priority
for system resources so that tasks do not impede HBase perf. Something like
YARN-443.
This JIRA is part of HADOOP-9165 but that one has been resolved as invalid.
> Dynamic resource configuration on NM
> ------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
