Li Lu commented on YARN-4265:

Another key addition in this patch, compare to YARN-3942, is a "cache id 
plugin" mechanism. One general problem the plugin needs to solve is "which 
exact storage should I look at on a read request like ...". The user can add a 
plugin (for example, a Tez plugin) in the configuration. At run time, on each 
read request, the plugin can help deciding if we need to look at the summary 
storage only or the detailed (entity) storage. In this way, the whole storage 
decision process of ATS 1.5 fine-grain cache storage will be:

- On writes, the application can decide if an entity should go to the summary 
storage or the detailed storage. If it should go to the latter, the application 
need to provide a cache id for this entity. 
- On reads, the application provides a plugin to decide if the look up can be 
performed on the summery storage, or on the detailed storage. If it's the 
latter, the plugin provides a cache id so that the storage should look at. 

> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>                 Key: YARN-4265
>                 URL: https://issues.apache.org/jira/browse/YARN-4265
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-4265-trunk.poc_001.patch
> To support the newly proposed APIs in YARN-4234, we need to create a new 
> plugin timeline store. The store may have similar behavior as the 
> EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id 
> granularity, instead of application id granularity. Let's have this storage 
> as a standalone one, instead of updating EntityFileTimelineStore, to keep the 
> existing store (EntityFileTimelineStore) stable. 

This message was sent by Atlassian JIRA

Reply via email to