[
https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886212#comment-13886212
]
Colin Patrick McCabe commented on HDFS-5851:
--------------------------------------------
Hi Arpit,
I don't know if you were present for some of the discussions around in-memory
caching and HDFS-4949. See
https://issues.apache.org/jira/browse/HDFS-4949?focusedCommentId=13707389&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13707389
for some discussion around this.
In the past, we've talked about having a "transient tier" for files that we
write, but don't necessarily want to put on-disk. I think many applications
would choose to write to a tier that would put stuff into memory if space was
available, but if not, would spill it to disk. It's crucial to implement
spilling, though. Otherwise, we make the applications worry about how much
memory is left on the DataNode, which I think would lead to limited adoption.
In this sense, memory gets used as a temporary area during a job, not so much a
"storage area" (at least that's how I look at it.) Does this line up with your
thinking in this area?
> Support memory as a storage medium
> ----------------------------------
>
> Key: HDFS-5851
> URL: https://issues.apache.org/jira/browse/HDFS-5851
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: 3.0.0
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
>
> Memory can be used as a storage medium for smaller/transient files for fast
> write throughput.
> More information/design will be added later.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
