[
https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981505#comment-14981505
]
Li Lu commented on YARN-4265:
-----------------------------
Another key addition in this patch, compare to YARN-3942, is a "cache id
plugin" mechanism. One general problem the plugin needs to solve is "which
exact storage should I look at on a read request like ...". The user can add a
plugin (for example, a Tez plugin) in the configuration. At run time, on each
read request, the plugin can help deciding if we need to look at the summary
storage only or the detailed (entity) storage. In this way, the whole storage
decision process of ATS 1.5 fine-grain cache storage will be:
- On writes, the application can decide if an entity should go to the summary
storage or the detailed storage. If it should go to the latter, the application
need to provide a cache id for this entity.
- On reads, the application provides a plugin to decide if the look up can be
performed on the summery storage, or on the detailed storage. If it's the
latter, the plugin provides a cache id so that the storage should look at.
> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>
> Key: YARN-4265
> URL: https://issues.apache.org/jira/browse/YARN-4265
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Li Lu
> Attachments: YARN-4265-trunk.poc_001.patch
>
>
> To support the newly proposed APIs in YARN-4234, we need to create a new
> plugin timeline store. The store may have similar behavior as the
> EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id
> granularity, instead of application id granularity. Let's have this storage
> as a standalone one, instead of updating EntityFileTimelineStore, to keep the
> existing store (EntityFileTimelineStore) stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
