Hitesh Shah commented on YARN-3942:

[~jlowe] [~rajesh.balamohan] observed that the timeline server was running out 
of memory in a certain scenario. In this scenario, we are using Hive-on-Tez but 
Hive re-uses the application to run 100s of DAGs/queries (doAs=false with 
perimeter security using say Ranger or Sentry). The EntityFileStore sizes a 
cache based on the no. of applications it can cache but in the above scenario, 
even a single app could be very large. Ideally, if each "dag" was in a separate 
file and all of its entries treated as a single cache entity - that would 
probably work better but making this generic enough may be a bit tricky.

Any suggestions here? 

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure 
> to mitigate some of the issues we've seen with ATS v1 while waiting for ATS 
> v2.  The intent of this plugin is to provide a workable solution for running 
> the Tez UI against the timeline server on a large-scale clusters running many 
> thousands of jobs per day.

This message was sent by Atlassian JIRA

Reply via email to