[
https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4949:
------------------------------
Attachment: caching-design-doc-2013-08-09.pdf
Suresh, thanks for posting your notes. Attached is a revised design doc that
beefs up the resource management / user quotas section, as well as addressing
your other smaller points.
As a meta-point, I think much of the remaining resource management design can
wait until after we get the initial end-to-end implementation going. I think
it's reasonable for the first iteration to do something simple like "superuser
only" or user quotas, then we layer on the complexities of pools, priorities,
ACLs, min/max/share, and failure cases afterwards. It's good to get the API
roughly right so we code with foresight, but I don't see us getting around to
implementing pools for at least a month or two.
> Centralized cache management in HDFS
> ------------------------------------
>
> Key: HDFS-4949
> URL: https://issues.apache.org/jira/browse/HDFS-4949
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: 3.0.0, 2.3.0
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: caching-design-doc-2013-07-02.pdf,
> caching-design-doc-2013-08-09.pdf
>
>
> HDFS currently has no support for managing or exposing in-memory caches at
> datanodes. This makes it harder for higher level application frameworks like
> Hive, Pig, and Impala to effectively use cluster memory, because they cannot
> explicitly cache important datasets or place their tasks for memory locality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
