[
https://issues.apache.org/jira/browse/HDFS-16128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408949#comment-17408949
]
Xing Lin commented on HDFS-16128:
---------------------------------
Hi [~prasad-acit],
The issue is given a inode as a long value, the function will first construct a
INode object. But we don't know what the parent Inode is for this INode, thus
we can not determine which partition to search for. That is why we fall back
to iterate over all partitions to search for that inode.
{code:java}
INode inode = new INodeDirectory(id, null,
new PermissionStatus("", "", new FsPermission((short) 0)), 0);{code}
You should also take a look at this function: public INode get(INode inode).
Inside this function, we first check whether there are KEY_DEPTH - 1 levels of
parent Inodes. If there are sufficient parent Inodes, then we go directly with
map.get(inode). Otherwise, we fall back to get(long inode), which basically
scan all partitions and search for the inode.
Hope this answers your question.
> [FGL] Add support for saving/loading an FS Image for PartitionedGSet
> --------------------------------------------------------------------
>
> Key: HDFS-16128
> URL: https://issues.apache.org/jira/browse/HDFS-16128
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, namenode
> Reporter: Xing Lin
> Assignee: Xing Lin
> Priority: Major
> Labels: pull-request-available
> Fix For: Fine-Grained Locking
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Add support to save Inodes stored in PartitionedGSet when saving an FS image
> and load Inodes into PartitionedGSet from a saved FS image.
> h1. Saving FSImage
> *Original HDFS design*: iterate every inode in inodeMap and save them into
> the FSImage file.
> *FGL*: no change is needed here, since PartitionedGSet also provides an
> iterator interface, to iterate over inodes stored in partitions.
> h1. Loading an HDFS
> *Original HDFS design*: it first loads the FSImage files and then loads edit
> logs for recent changes. FSImage files contain different sections, including
> INodeSections and INodeDirectorySections. An InodeSection contains serialized
> Inodes objects and the INodeDirectorySection contains the parent inode for an
> Inode. When loading an FSImage, the system first loads INodeSections and then
> load the INodeDirectorySections, to set the parent inode for each inode.
> After FSImage files are loaded, edit logs are then loaded. Edit log contains
> recent changes to the filesystem, including Inodes creation/deletion. For a
> newly created INode, the parent inode is set before it is added to the
> inodeMap.
> *FGL*: when adding an Inode into the partitionedGSet, we need the parent
> inode of an inode, in order to determine which partition to store that inode,
> when NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we
> used a temporary LightweightGSet (inodeMapTemp), to store inodes. When
> LoadFSImage is done, the parent inode for all existing inodes in FSImage
> files is set. We can now move the inodes into a partitionedGSet. Load edit
> logs can work as usual, as the parent inode for an inode is set before it is
> added to the inodeMap.
> In theory, PartitionedGSet can support to store inodes without setting its
> parent inodes. All these inodes will be stored in the 0th partition. However,
> we decide to use a temporary LightweightGSet (inodeMapTemp) to store these
> inodes, to make this case more transparent.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
