[
https://issues.apache.org/jira/browse/HDFS-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563531#comment-14563531
]
Arpit Agarwal commented on HDFS-8401:
-------------------------------------
bq. The administrator simply set a list of files and directories to be cached.
When applications read those files or directories, they were retrieved from the
cache.
It's impractical to involve the administrator every time a new file is to be
cached. We've heard this requirement makes caching difficult to use. There are
a couple of other things can help with usability e.g. de-duplication of cache
directives, predictability of cache locality.
bq. If we just want a generic FS caching layer in Hadoop, we could do that in
hadoop-common.
That was my intention. I'll move the jira to common.
> Memfs - a layered file system for in-memory storage in HDFS
> -----------------------------------------------------------
>
> Key: HDFS-8401
> URL: https://issues.apache.org/jira/browse/HDFS-8401
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
>
> We propose creating a layered filesystem that can provide in-memory storage
> using existing features within HDFS. memfs will use lazy persist writes
> introduced by HDFS-6581. For reads, memfs can use the Centralized Cache
> Management feature introduced in HDFS-4949 to load hot data to memory.
> Paths in memfs and hdfs will correspond 1:1 so memfs will require no
> additional metadata and it can be implemented entirely as a client-side
> library.
> The advantage of a layered file system is that it requires little or no
> changes to existing applications. e.g. Applications can use something like
> {{memfs://}} instead of {{hdfs://}} for files targeted to memory storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
