Konstantinos Karanasos commented on YARN-2884:

[~kasha], [~curino], [~subru], given that this proxy/agent will only focus on 
the AM-RM communication, we may also explicitly call it AMRMProxy or AMRMAgent 
(following the naming convention of the already existing AMRMClient* classes).

[~djp] I just added a comment in the umbrella JIRA (YARN-2877), trying to give 
some more details.
We are not proposing to substitute all scheduling decisions with distributed 
ones. The guaranteed-start containers will continue to be scheduled by the 
central RM. However, the queueable ones will be scheduled in a distributed 
The first candidate for queueable containers is the short-running tasks, in 
which the overhead of contacting the central RM is a significant part of the 
overall task execution time. Scheduling these requests without contacting the 
central RM will reduce their latency, increase the utilization of the cluster 
(no idle resources waiting to contact the RM), while it will offload the 
central RM (which is good for scaling in big clusters).

> Proxying all AM-RM communications
> ---------------------------------
>                 Key: YARN-2884
>                 URL: https://issues.apache.org/jira/browse/YARN-2884
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Carlo Curino
> We introduce the notion of an RMProxy, running on each node (or once per 
> rack). Upon start the AM is forced (via tokens and configuration) to direct 
> all its requests to a new services running on the NM that provide a proxy to 
> the central RM. 
> This give us a place to:
> 1) perform distributed scheduling decisions
> 2) throttling mis-behaving AMs
> 3) mask the access to a federation of RMs

This message was sent by Atlassian JIRA

Reply via email to