[
https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707536#comment-13707536
]
Suresh Srinivas commented on HDFS-4949:
---------------------------------------
bq. However, tiered storage management schemes put an entire block into a tier.
This is different than what we want to (eventually) do with caching, which is
cache only part of a block. If we end up with a heavyweight scheme where the
entire block has to be loaded into memory before any of it can be accessed,
this may actually cause a performance regression, not an improvement.
This is one point I have been mulling over as well. I agree, partially cached
data does not fit with storage hierarchy. But I was not sure partial cache is
being consider in the first phase of implementation in this jira (sorry I am
yet to read this carefully, If i missed details).
One other thing that we are considering is usage based extra replica to memory
tier. Need to make some time to get that all into a doc.
> Centralized cache management in HDFS
> ------------------------------------
>
> Key: HDFS-4949
> URL: https://issues.apache.org/jira/browse/HDFS-4949
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: 3.0.0, 2.2.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: caching-design-doc-2013-07-02.pdf
>
>
> HDFS currently has no support for managing or exposing in-memory caches at
> datanodes. This makes it harder for higher level application frameworks like
> Hive, Pig, and Impala to effectively use cluster memory, because they cannot
> explicitly cache important datasets or place their tasks for memory locality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
