[
https://issues.apache.org/jira/browse/HDFS-8286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083750#comment-15083750
]
Haohui Mai commented on HDFS-8286:
----------------------------------
I have pushed the prototype that corresponds to my [Hadoop summit
talk|http://www.slideshare.net/HaohuiMai/partial-nshadoopsummit2015] to the
[feature-HDFS-8286|https://github.com/apache/hadoop/tree/feature-HDFS-8286]
branch.
> Scaling out the namespace using KV store
> ----------------------------------------
>
> Key: HDFS-8286
> URL: https://issues.apache.org/jira/browse/HDFS-8286
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Haohui Mai
> Assignee: Haohui Mai
> Attachments: hdfs-kv-design.pdf
>
>
> Currently the NN keeps the namespace in the memory. To improve the
> scalability of the namespace, users can scale up by using more RAM or scale
> out using Federation (i.e., statically partitioning the namespace).
> We would like to remove the limitation of scaling the global namespace. Our
> vision is that that HDFS should adopt a scalable underlying architecture that
> allows the global namespace scales linearly.
> We propose to implement the HDFS namespace on top of a key-value (KV) store.
> Adopting the KV store interfaces allows HDFS to leverage the capability of
> modern KV store and to become much easier to scale. Going forward, the
> architecture allows distributing the namespace across multiple machines, or
> storing only the working set in the memory (HDFS-5389), both of which allows
> HDFS to manage billions of files using the commodity hardware available today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
