[
https://issues.apache.org/jira/browse/YARN-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500894#comment-14500894
]
Inigo Goiri commented on YARN-3482:
-----------------------------------
Hi Sunil G, yes, I'm talking about Total CPU and Total Memory.
Combining this with YARN-3481, we can estimate the load in the node that is not
caused by the containers (external processes).
Right now, the server could be overloaded by HBase for example and we would be
sending more load there.
As Karthik Kambatla mentions, this would be a very conservative scenario where
the external processes have absolute priority.
This might be a desired behavior for some users but the proposal is to also add
an interface to dynamically change the amount of available resources according
to the behavior of the external processes.
Both approaches target the same problem and are complementary/orthogonal.
I understand this other approach of sending node utilization might be a little
out of the scope of this JIRA but I could open a new one with this
functionality.
> Report NM available resources in heartbeat
> ------------------------------------------
>
> Key: YARN-3482
> URL: https://issues.apache.org/jira/browse/YARN-3482
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager
> Affects Versions: 2.7.0
> Reporter: Inigo Goiri
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> NMs are usually collocated with other processes like HDFS, Impala or HBase.
> To manage this scenario correctly, YARN should be aware of the actual
> available resources. The proposal is to have an interface to dynamically
> change the available resources and report this to the RM in every heartbeat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
