Jason Lowe commented on YARN-3942:

bq.  if you'd like to elaborate the drawback a bit, it will be helpful.

Yes the drawback is that one cannot do cross-application queries unless the 
entity type is stored in the main database (i.e.: listed in 
yarn.timeline-service.entity-file-store.summary-entity-types).  Another case 
that wouldn't work is where the query has multiple application IDs in it -- the 
query processing will choose the HDFS store of one of the applications and fail 
to find entities for the others.  In practice the Tez UI only does 
cross-application queries on the "All DAGs" front page, and that only needs a 
small amount of entity types as I listed above.  Since that's the main use case 
we're optimizing for with this approach, it allows us to offload most of the 
entity types from the leveldb database and serve them directly from HDFS.

So this solution is a tradeoff.  It limits the types of queries that the 
timeline server can properly answer but significantly scales the single-node 
timeline server and decouples the jobs posting events from the timeline server. 
 The latter is particularly interesting for us, as we no longer have a 
mission-critical single node for running jobs that need to post timeline events.

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure 
> to mitigate some of the issues we've seen with ATS v1 while waiting for ATS 
> v2.  The intent of this plugin is to provide a workable solution for running 
> the Tez UI against the timeline server on a large-scale clusters running many 
> thousands of jobs per day.

This message was sent by Atlassian JIRA

Reply via email to