Vrushali C updated YARN-3167:
    Attachment: RM-AM-NM-Aggregator.png

Attaching a couple of diagrams that illustrate the design discussions between 
me and Sangjin and Karthik and Robert.

Some notes:
- Per node aggregator process runs as an auxillary service inside the NM
- It has a per application datastructure to receive and write entity info for 
that app
- The writer API has the following (at the least): 
---- write (entity) : this will write all the contents of the entity to the 
backend store
---- addEvent (application id, event) : this writes the lifecycle events
---- updateMetrics (application id, timeline metric) : this writes out the 
- the base aggregator service api also has the above in its API
- in addition to these, the base aggregator service could do a sync/async write 
call for each of the above. 

For the PoC, we could choose to have every write as a synchronous write.

> [Aggregator implementation] Implement the core functionality of the base 
> aggregator service
> -------------------------------------------------------------------------------------------
>                 Key: YARN-3167
>                 URL: https://issues.apache.org/jira/browse/YARN-3167
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: RM-AM-NM-Aggregator.png, 
> Sequence_diagram_User_RM_AM_NM_Aggregator_Writer.png
> The basic skeleton of the timeline aggregator has been set up by YARN-3030. 
> We need to implement the core functionality of the base aggregator service. 
> The key things include
> - handling the requests from clients (sync or async)
> - buffering data
> - handling the aggregation logic
> - invoking the storage API

This message was sent by Atlassian JIRA

Reply via email to