Junping Du commented on YARN-3033:

Thanks [~gtCarrera9] for delivering a proposal here which looks pretty good. A 
couple of comments here:
bq. In the below discussion, we start with a notion of AMs/NMs not making any 
assumptions on the physical locations of the aggregators an
aggregator can sit next to the AM in a separate container, it can side on the 
local node etc.
I think we should mention our best-effort to place logical app aggregator to be 
on the same physical node with AM containers, given the assumption that AM 
could consume aggregator service heavier so we want to keep data transferred in 
local as much as possible. The other cases that AM don't stay with logical 
aggregator is AM get failed over to other node but NM physical aggregator 
service is still alive so logic aggregator keep the same place. We'd better to 
mention here.

bq. Depending on the number of running applications ( N_app ) and the number of 
nodes ( N_node ) or on the size of individual applications, we may choose 
different mapping policies (implemented as different aggregatorcollections).
For the reason I mentioned above, we should do our best effort to keep logical 
aggregator stay with AM container and we don't have to have different policies 
for different cases. Even for the case N_app >> N_node, we should make sure AMs 
doesn't get skewed (aggregated) across the cluster.

 Also I didn't see the discussion on launching (or we call binding - to 
physical aggregator) the logical aggregator (by who and how). I assume this is 
the scope of this JIRA. Isn't it?

> [Aggregator wireup] Implement NM starting the ATS writer companion
> ------------------------------------------------------------------
>                 Key: YARN-3033
>                 URL: https://issues.apache.org/jira/browse/YARN-3033
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Li Lu
>         Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf
> Per design in YARN-2928, implement node managers starting the ATS writer 
> companion.

This message was sent by Atlassian JIRA

Reply via email to