[
https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881444#action_12881444
]
Konstantin Shvachko commented on HDFS-1071:
-------------------------------------------
> saveNamespace can only be done when in safemode, so we are only performing
> read operations on the datastructure that is essentially read only
Yes you can saveNamespace only only in safe mode, but the data structures are
not read only. For example, {{getBlockLocations()}} updates {{accessTime}} for
file inodes. This means that different threads in your implementation may
record different atimes of the same file, depending on when they write it. This
means that file system images will not be identical in different directories.
They have been so far.
> while the parent thread is holding a lock, right?
Are you relying on the parent thread lock? Could you please explain how that
works?
In fact I thought you would implement something like this: one thread traverses
the tree, serializes image objects and puts them into a queue, where other
writing threads pick them up and write to disk in parallel. Then it is
guaranteed that images are exactly the same.
> so it will take 1.5-2 minutes for both.
Have you actually tested it? Is it the same if both directories are on the same
drive?
> savenamespace should write the fsimage to all configured fs.name.dir in
> parallel
> --------------------------------------------------------------------------------
>
> Key: HDFS-1071
> URL: https://issues.apache.org/jira/browse/HDFS-1071
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.4.patch,
> HDFS-1071.5.patch, HDFS-1071.patch
>
>
> If you have a large number of files in HDFS, the fsimage file is very big.
> When the namenode restarts, it writes a copy of the fsimage to all
> directories configured in fs.name.dir. This takes a long time, especially if
> there are many directories in fs.name.dir. Make the NN write the fsimage to
> all these directories in parallel.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
