[
https://issues.apache.org/jira/browse/HDFS-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860755#comment-13860755
]
Suresh Srinivas commented on HDFS-5711:
---------------------------------------
[~rohanpasalkar], few comments:
- Can you change the title of this jira appropriately. It looks like your
intent was to also create a jira for persisting block locations on disk, which
should be a subtask of this umbrella jira?
- The title still may be not appropriate to the subtask. Even if you persist
block information (which is a small part of namenode memory resident data),
there is still a lot of information in NN memory.
I have hard time understanding creating an umbrella jira that tracks too many
things. Performance and responsiveness should be separate from scalability. The
former is already captured in component "Performance". Why create an umbrella
jira for that? Feel free to add Performance into the component of relevant
jiras.
Also are you going to post a design document and work on things you are
proposing?
> Removing memory limitation of the Namenode by persisting Block - Block
> location mappings to disk.
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-5711
> URL: https://issues.apache.org/jira/browse/HDFS-5711
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Rohan Pasalkar
>
> This jira is to track changes to be made to remove HDFS name-node memory
> limitation to hold block - block location mappings.
> It is a known fact that the single Name-node architecture of HDFS has
> scalability limits. The HDFS federation project alleviates this problem by
> using horizontal scaling. This helps increase the throughput of metadata
> operation and also the amount of data that can be stored in a Hadoop cluster.
> The Name-node stores all the filesystem metadata in memory (even in the
> federated architecture), the
> Name-node design can be enhanced by persisting part of the metadata onto
> secondary storage and retaining
> the popular or recently accessed metadata information in main memory. This
> design can benefit a HDFS deployment
> which doesn't use federation but needs to store a large number of files or
> large number of blocks. Lin Xiao from Hortonworks attempted a similar
> project [1] in the Summer of 2013. They used LevelDB to persist the Namespace
> information (i.e file and directory inode information).
> A patch with this change is yet to be submitted to code base. We also intend
> to use LevelDB to persist metadata, and plan to
> provide a complete solution, by not just persisting the Namespace
> information but also the Blocks Map onto secondary storage.
> We did implement the basic prototype which stores the block-block location
> mapping metadata to the persistent key-value store i.e. levelDB. Prototype
> also maintains the in-memory cache of the recently used block-block location
> mappings metadata.
> References:
> [1] Lin Xiao, Hortonworks, Removing Name-node’s memory limitation,
> http://www.slideshare.net/ydn/hadoop-meetup-hug-august-2013-removing-the-namenodes-memory-limitation
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
