Li Lu updated YARN-3942:
    Attachment: YARN-3942-leveldb.002.patch

Thanks [~gss2002]! I modified the patch with your changes. As a quick update, 
the current Leveldb cache storage patch addresses the (potential) OOM issue. 
However, now we inevitably have to load all entities as a whole when refreshing 
the cache, which will introduce longer latency. I discussed this with [~hitesh] 
and [~xgong] and seems like the best solution to the latency problem will be 
reducing the granularity of the caches. Instead of loading (and caching) 
entities for a whole app, we need to load and cache a set of entities, like 
those ones in the same Tez DAG. Let's focus on the current fix in this JIRA, 
and open a new JIRA for the next step? Comments and suggestions are more than 

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942-leveldb.001.patch, 
> YARN-3942-leveldb.002.patch, YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure 
> to mitigate some of the issues we've seen with ATS v1 while waiting for ATS 
> v2.  The intent of this plugin is to provide a workable solution for running 
> the Tez UI against the timeline server on a large-scale clusters running many 
> thousands of jobs per day.

This message was sent by Atlassian JIRA

Reply via email to