[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619059#comment-13619059
]
Alejandro Abdelnur commented on YARN-291:
-----------------------------------------
+1 on the idea of dynamic resource configuration. The sharing w/HBase use case
is a scenario we see quite often.
Though, I have concerns on changing the available NMs capacity directly in the
NMs. This would complicate significantly allocation changes/re-computation of
resources for running applications.
NM available resources discovery should be static as it is completely
acceptable to restart them without compromising running apps:
* A NM on startup detects the OS for its CPU/memory and reports this to the RM
** config properties in the NM, if present, could override OS detection values
* A NM should be restarted to report a new configuration
To achieve dynamic resource configuration we should fully leverage YARN
resource scheduling framework. For example, this would be done by introducing
the concept of 'unmanaged resources'. The idea would be something like this:
* An (unmanaged) AM requests unmanaged resources from the RM (a new flag in a
resource request)
* The RM assigns the unmanaged resource to a NM
* The RM notifies the NM of the unmanaged resource assignment
* The RM notifies the AM of the unmanaged resource assignment
* The unmanaged assigned resource does not time out in the RM/scheduler due to
container not starting (Bikas, our chat in the last yarn meet-up clicked)
* The AM will, out of band, claim those resources from the corresponding node
* The resources will remain assigned to the AM until the AM releases them, the
AM goes MIA or the resources are preempted
This together with YARN-392 (enforce locality for a request), will allow an
external component to claim cluster resources while enforcing existing YARN
resource allocation policies and priorities.
With this approach the changes in the RM/scheduler and API are quite simple.
Thoughts?
> Dynamic resource configuration on NM
> ------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch,
> YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> About more background and design details, please refer: HADOOP-9165.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
