[
https://issues.apache.org/jira/browse/HDDS-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147817#comment-17147817
]
Yiqun Lin edited comment on HDDS-2939 at 6/29/20, 2:43 PM:
-----------------------------------------------------------
Hi [~maobaolong], looked into the approach that used in Alluxio, I am curious
one thing: In Alluxio 2.0, it still keeps all metadata in its master service?
I see this in article
[https://dzone.com/articles/store-1-billion-files-in-alluxio-20] that describes
how to store 1 billion files metadata in Alluxio.
{quote}The metadata service in Alluxio 2.0 is designed to support at least 1
billion files with a significantly reduced memory requirement. To achieve this,
we added support for storing part of the namespace off-heap by RocksDB on disk.
Recently-accessed file system metadata is stored in memory,...
{quote}
>From my understanding of this, in disk only part of metadata stored and
>meanwhile in memory it caches recent-accessed data, does it really store all
>metadata in its master service? does it match the case of Ozone Fs? In Ozone
>Fs, we will store all metadata. Or can I understand that Alluxio 2.0
>maintains only active metadata instead of the whole metadata, this active
>metadata can be updated(activated/deactivated) by the user access file
>behaviors, so it can support billion level metadata.
BTW, memory cached only for hot metadata is a good point that Ozone Fs can also
benefit from this.
was (Author: linyiqun):
Hi [~maobaolong], looked into the approach that used in Alluxio, I am curious
one thing: In Alluxio 2.0, it still keeps all metadata in its master service?
I see this in article
[https://dzone.com/articles/store-1-billion-files-in-alluxio-20] that describes
how to store 1 billion files metadata in Alluxio.
{quote}The metadata service in Alluxio 2.0 is designed to support at least 1
billion files with a significantly reduced memory requirement. To achieve this,
we added support for storing part of the namespace off-heap by RocksDB on disk.
Recently-accessed file system metadata is stored in memory,...
{quote}
>From my understanding of this, in disk only part of metadata stored and
>meanwhile in memory it caches recent-accessed data, does it really store all
>metadata in its master service? does it match the case of Ozone Fs? In Ozone
>Fs, we will store all metadata.
BTW, memory cached only for hot metadata is a good point that Ozone Fs can also
benefit from this.
> Ozone FS namespace
> ------------------
>
> Key: HDDS-2939
> URL: https://issues.apache.org/jira/browse/HDDS-2939
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Manager
> Reporter: Supratim Deka
> Assignee: Rakesh Radhakrishnan
> Priority: Major
> Labels: Triaged
> Attachments: Ozone FS Namespace Proposal v1.0.docx
>
>
> Create the structures and metadata layout required to support efficient FS
> namespace operations in Ozone - operations involving folders/directories
> required to support the Hadoop compatible Filesystem interface.
> The details are described in the attached document. The work is divided up
> into sub-tasks as per the task list in the document.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
