[
https://issues.apache.org/jira/browse/YARN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221025#comment-14221025
]
Konstantinos Karanasos commented on YARN-2884:
----------------------------------------------
[~kasha], [~curino], [~subru], given that this proxy/agent will only focus on
the AM-RM communication, we may also explicitly call it AMRMProxy or AMRMAgent
(following the naming convention of the already existing AMRMClient* classes).
[~djp] I just added a comment in the umbrella JIRA (YARN-2877), trying to give
some more details.
We are not proposing to substitute all scheduling decisions with distributed
ones. The guaranteed-start containers will continue to be scheduled by the
central RM. However, the queueable ones will be scheduled in a distributed
fashion.
The first candidate for queueable containers is the short-running tasks, in
which the overhead of contacting the central RM is a significant part of the
overall task execution time. Scheduling these requests without contacting the
central RM will reduce their latency, increase the utilization of the cluster
(no idle resources waiting to contact the RM), while it will offload the
central RM (which is good for scaling in big clusters).
> Proxying all AM-RM communications
> ---------------------------------
>
> Key: YARN-2884
> URL: https://issues.apache.org/jira/browse/YARN-2884
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Carlo Curino
>
> We introduce the notion of an RMProxy, running on each node (or once per
> rack). Upon start the AM is forced (via tokens and configuration) to direct
> all its requests to a new services running on the NM that provide a proxy to
> the central RM.
> This give us a place to:
> 1) perform distributed scheduling decisions
> 2) throttling mis-behaving AMs
> 3) mask the access to a federation of RMs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
