[
https://issues.apache.org/jira/browse/HDFS-8286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623205#comment-14623205
]
Yong Zhang commented on HDFS-8286:
----------------------------------
If HDFS keep metadata in a persistent KV store, such as leveldb, is fsimage
necessary?
> Scaling out the namespace using KV store
> ----------------------------------------
>
> Key: HDFS-8286
> URL: https://issues.apache.org/jira/browse/HDFS-8286
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Haohui Mai
> Assignee: Haohui Mai
> Attachments: hdfs-kv-design.pdf
>
>
> Currently the NN keeps the namespace in the memory. To improve the
> scalability of the namespace, users can scale up by using more RAM or scale
> out using Federation (i.e., statically partitioning the namespace).
> We would like to remove the limitation of scaling the global namespace. Our
> vision is that that HDFS should adopt a scalable underlying architecture that
> allows the global namespace scales linearly.
> We propose to implement the HDFS namespace on top of a key-value (KV) store.
> Adopting the KV store interfaces allows HDFS to leverage the capability of
> modern KV store and to become much easier to scale. Going forward, the
> architecture allows distributing the namespace across multiple machines, or
> storing only the working set in the memory (HDFS-5389), both of which allows
> HDFS to manage billions of files using the commodity hardware available today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
