Sangjin Lee commented on YARN-3034:
Sorry for my delayed response...
As for the code organization, also see the discussion on YARN-3166.
If we plan to handle similar to current approach i.e send the Entity data
through a rest client to a timeline writer service(RMTimelineAggregator), where
should this service be running i.e. as part of which process or should it be a
daemon on its own?
Since this is going to be exclusively used by the resource manager, we have two
options: (1) start it inside the RM process as a service, or (2) start it as a
standalone daemon on the RM machine. I think either is fine, although it would
be good to both as options.
Is RMTimelineAggregator is expected to do any primary (preliminary) aggregation
of some metrics ? Just wanted to the know reason to have a specific
TimeLineAggregator for RM separately?
I do think the RM will simply push data directly to the timeline storage
without much aggregation, and so on. The RM doesn’t really handle the app-level
aggregation or separation (that’s why the RM’s aggregator service needs to
extend the BaseAggregatorService directly).
IIUC RM needs to add User and Queue Entities when application is created if the
specified user and queue doesnt exist as entity in ATS ? Apart from this Queue
Entity has Parent Queue information, is it something like when CS/FS is
initialized we need to create Entities for new queues and hierarcies ? Is it
not sufficent to just have for Leaf Queue Entity and just have parent path as
its meta info, is hierarchy req?
I would say let’s not worry about the queue and the user from the RM point of
view. As discussed in YARN-2928, the user and queue entity classes are there
primarily for serving reads.
In the original design, it is proposed that we need to organize application
level aggregators into collections (either on the NMs or on the RM, supposedly
implemented as AppLevelServiceManager? ), and the servers launches its own
As mentioned above, the RM doesn’t really need to deal with the per-app
boundary. For a practical perspective also, if it did it could become a fairly
big memory hotspot if the RM’s aggregator starts collecting per-app stuff as it
deals with a large number of applications. We envision the RM aggregator as
pushing entities pretty directly onto storage.
> [Aggregator wireup] Implement RM starting its ATS writer
> Key: YARN-3034
> URL: https://issues.apache.org/jira/browse/YARN-3034
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Naganarasimha G R
> Attachments: YARN-3034.20150205-1.patch
> Per design in YARN-2928, implement resource managers starting their own ATS
This message was sent by Atlassian JIRA