Jason Lowe commented on YARN-3942:

Yeah that's going to be tricky, especially if we need to move most of the code 
into YARN.  Haven't had time to give this much thought, but the only way I can 
think of to keep most of the functionality in YARN is to have the timeline 
client be able to specify when a new "session" starts (i.e.: entity file writer 
should start writing to a new file and user provides some clue/hint as to what 
to name the file).  We can then have a plugin on the entity file server side 
that allows apps to override the getTimelineStoreForRead functionality.

If that was in place then the Tez side could start a new session (dag file) 
each time the dag changed.  The Tez-specific plugin on the timeline server side 
could then translate dag/vertex/task/attempt IDs into the appropriate dag file 
to cache.  There would still be some questions as to how the timeline store 
cache would be managed on the server side and how to support multiple 
framework-specific plugins simultaneously.

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure 
> to mitigate some of the issues we've seen with ATS v1 while waiting for ATS 
> v2.  The intent of this plugin is to provide a workable solution for running 
> the Tez UI against the timeline server on a large-scale clusters running many 
> thousands of jobs per day.

This message was sent by Atlassian JIRA

Reply via email to