[
https://issues.apache.org/jira/browse/HDDS-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162743#comment-17162743
]
Rakesh Radhakrishnan edited comment on HDDS-2939 at 7/22/20, 12:14 PM:
-----------------------------------------------------------------------
Thanks a lot [~maobaolong] , [~linyiqun] for pointing to the Alluxio metadata
structure. Sure, will consider these as a referral docs.
I have created [HDDS-2939
branch|https://github.com/apache/hadoop-ozone/tree/HDDS-2939] for the feature
development and posted first
[PR-1230|https://github.com/apache/hadoop-ozone/pull/1230] to implement create
directory #mkdirs user call. In this patch, I have tried using the existing
'PrefixTable' for storing the dirs. I am also exploring the necessity of a
separate new DirTable instead of PrefixTable, which currently used only for
keeping the ACLs. Welcome comments, thanks!
There is a plan to maintain an in-memory dir/key cache to avoid the look up
overhead and needs to understand the size of this cache. Basically, for every
dir lookup, it has to seek the dirs in 'DirTable'. For every file look up, it
has to seek all the parents upto the immediate parent dir in 'DirTable' and
then seek the file key in the 'KeyTable'.
Below tables shows the existing key structure and the proposed idea. The table
content is just a representation to show the internals.
*1) User given Paths under /vol1/buck1:*
|{color:#0747a6}*Type*{color}|{color:#0747a6}*Path*{color}|
|File|/a/b/c/d/file1|
|Dir|/a/b/c/d/e|
|File|/a/b/c/d/file2|
|Dir|/a/b/c/d/e/file1|
*2) Existing KeyTable Data Format:*
|{color:#00875a}*KeyName*{color}|
|/vol1/buck1/a/b/c/d/file1|
|/vol1/buck1/a/b/c/d/e/|
|/vol1/buck1/a/b/c/d/file2|
|/vol1/buck1/a/b/c/d/e/file1/|
*3) Proposed data format -table Dir/File entry: "parentID/Key".* Assume bucket
ObjectID is 512, which is the parent for the first level dirs/files
*PrefixTable:*
|{color:#ff8b00}*PrefixName*{color}|{color:#ff8b00}*ObjectID*{color}|
|512/a|1025|
|1025/b|1026|
|1026/c|1027|
|1027/d|1028|
|1028/e|1029|
|1029/file1|1032|
*KeyTable:*
|{color:#ff8b00}*KeyName*{color}|{color:#ff8b00}*ObjectID*{color}|
|1028/file1|1030|
|1028/file2|1031|
was (Author: rakeshr):
Thanks a lot [~maobaolong] , [~linyiqun] for pointing to the Alluxio metadata
structure. Sure, will consider these as a referral docs.
I have created [HDDS-2939
branch|https://github.com/apache/hadoop-ozone/tree/HDDS-2939] for the feature
development and posted first
[PR-1230|https://github.com/apache/hadoop-ozone/pull/1230] to implement create
directory #mkdirs user call. In this patch, I have tried using the existing
'PrefixTable' for storing the dirs. I am also exploring the necessity of a
separate new DirTable instead of PrefixTable, which currently used only for
keeping the ACLs. Welcome comments, thanks!
There is a plan to maintain an in-memory dir/key cache to avoid the look up
overhead and needs to understand the size of this cache. Basically, for every
dir lookup, it has to seek the dirs in 'DirTable'. For every file look up, it
has to seek all the parents upto the immediate parent dir in 'DirTable' and
then seek the file key in the 'KeyTable'.
Below tables shows the existing key structure and the proposed idea.
| {color:#00875a}
*Existing KeyTable Data Format*{color}|
| *Type*| *Path*| *KeyTable
Entry*|
|File|/a/b/c/d/file1|/vol1/buck1/a/b/c/d/file1|
|Dir|/a/b/c/d/e|/vol1/buck1/a/b/c/d/e/|
|File|/a/b/c/d/file2|/vol1/buck1/a/b/c/d/file2|
|Dir|/a/b/c/d/e/file1|/vol1/buck1/a/b/c/d/e/file1/|
| *{color:#ff8b00} Proposed data format -table Dir/File entry:
"parentID/Key"{color}*|
|{color:#57d9a3} *Bucket ObjectID - 512, which is the parent for the first
level dirs/files*{color}|
|{color:#0747a6}*PrefixTable - Data Format:
'parentID/Key'*{color}|{color:#0747a6}*KeyTable - Data Format:
'parentID/Key'*{color}|
| *PrefixName*| *ObjectID*| *KeyName*| *ObjectID*|
|512/a|1025|1028/file1|1030|
|1025/b|1026|1028/file2|1031|
|1026/c|1027| | |
|1027/d|1028| | |
|1028/e|1029| | |
|1029/file1|1032| | |
> Ozone FS namespace
> ------------------
>
> Key: HDDS-2939
> URL: https://issues.apache.org/jira/browse/HDDS-2939
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Manager
> Reporter: Supratim Deka
> Assignee: Rakesh Radhakrishnan
> Priority: Major
> Labels: Triaged
> Attachments: Ozone FS Namespace Proposal v1.0.docx
>
>
> Create the structures and metadata layout required to support efficient FS
> namespace operations in Ozone - operations involving folders/directories
> required to support the Hadoop compatible Filesystem interface.
> The details are described in the attached document. The work is divided up
> into sub-tasks as per the task list in the document.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
