[
https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973462#comment-13973462
]
Andrew Wang commented on HDFS-5851:
-----------------------------------
I'd really like to integrate this with HDFS-4949 where possible. One concern is
that we should avoid having another pool of memory carved off from the cluster.
HDFS-4949's cache pools were designed to eventually integrate with YARN, but
this might introduce another separate pool for a memory quota, putting us back
in the same place.
There are also plans to move towards sub-block caching. Whole-block caching is
wasteful for columnar formats like ORC and Parquet. With sub-block caching,
automatic cache replacement looks a lot more attractive (another planned
feature). These are both things we can support with HDFS-4949's infrastructure.
I'm not sure about with HSM.
It'd also be nice if apps could ZCR these memory-only replicas, ideally reusing
the existing auto-ZCR infrastructure.
Anyway, either pulling this towards HDFS-4949 or vice versa, we should figure
out these details before moving ahead. I'll echo Colin's desire for a meeting
to discuss this. We're willing to host at our Palo Alto office.
> Support memory as a storage medium
> ----------------------------------
>
> Key: HDFS-5851
> URL: https://issues.apache.org/jira/browse/HDFS-5851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
>
> Memory can be used as a storage medium for smaller/transient files for fast
> write throughput.
> More information/design will be added later.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
