[ 
https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729843#comment-14729843
 ] 

Jason Lowe commented on YARN-3942:
----------------------------------

Option 1 will add some latency (not clear how much yet) to initializing the 
cache, and it could take quite a bit of time to build it depending upon how 
many dags were run in the same session and the amount of data from each dag.

If I understand option 2 properly, it proposes to have the scanner read all the 
data, not just the summary data, out of HDFS and store it in the main leveldb.  
The problem we run into with that approach is that for our production scale and 
desired retention periods it would generate a very, very large set of leveldb 
databases that must be stored locally, and query performance starts to degrade 
as the leveldb databases get really large.

Option 1 is more viable for us, assuming we won't have horrendous latency 
issues trying to build a substantial database from a monster session.  Option 2 
is not as attractive, although I could see it being appealing to those that 
don't need to worry about huge leveldb size problems.

> Timeline store to read events from HDFS
> ---------------------------------------
>
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
>
>
> This adds a new timeline store plugin that is intended as a stop-gap measure 
> to mitigate some of the issues we've seen with ATS v1 while waiting for ATS 
> v2.  The intent of this plugin is to provide a workable solution for running 
> the Tez UI against the timeline server on a large-scale clusters running many 
> thousands of jobs per day.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to