Sangjin Lee commented on YARN-3034:

Sorry for my delayed response...

As for the code organization, also see the discussion on YARN-3166.

If we plan to handle similar to current approach i.e send the Entity data 
through a rest client to a timeline writer service(RMTimelineAggregator), where 
should this service be running i.e. as part of which process or should it be a 
daemon on its own?
Since this is going to be exclusively used by the resource manager, we have two 
options: (1) start it inside the RM process as a service, or (2) start it as a 
standalone daemon on the RM machine. I think either is fine, although it would 
be good to both as options.

Is RMTimelineAggregator is expected to do any primary (preliminary) aggregation 
of some metrics ? Just wanted to the know reason to have a specific 
TimeLineAggregator for RM separately?
I do think the RM will simply push data directly to the timeline storage 
without much aggregation, and so on. The RM doesn’t really handle the app-level 
aggregation or separation (that’s why the RM’s aggregator service needs to 
extend the BaseAggregatorService directly).

IIUC RM needs to add User and Queue Entities when application is created if the 
specified user and queue doesnt exist as entity in ATS ? Apart from this Queue 
Entity has Parent Queue information, is it something like when CS/FS is 
initialized we need to create Entities for new queues and hierarcies ? Is it 
not sufficent to just have for Leaf Queue Entity and just have parent path as 
its meta info, is hierarchy req?
I would say let’s not worry about the queue and the user from the RM point of 
view. As discussed in YARN-2928, the user and queue entity classes are there 
primarily for serving reads.

In the original design, it is proposed that we need to organize application 
level aggregators into collections (either on the NMs or on the RM, supposedly 
implemented as AppLevelServiceManager? ), and the servers launches its own 
As mentioned above, the RM doesn’t really need to deal with the per-app 
boundary. For a practical perspective also, if it did it could become a fairly 
big memory hotspot if the RM’s aggregator starts collecting per-app stuff as it 
deals with a large number of applications. We envision the RM aggregator as 
pushing entities pretty directly onto storage.

> [Aggregator wireup] Implement RM starting its ATS writer
> --------------------------------------------------------
>                 Key: YARN-3034
>                 URL: https://issues.apache.org/jira/browse/YARN-3034
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3034.20150205-1.patch
> Per design in YARN-2928, implement resource managers starting their own ATS 
> writers.

This message was sent by Atlassian JIRA

Reply via email to