[
https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872058#action_12872058
]
Konstantin Shvachko commented on HDFS-1071:
-------------------------------------------
# FSImage.processIOError() and FSEditsLog.processIOError() go as a pair, sort
of. If you change prototype of one you should also change the other's. So
{{FSEditsLog.processIOError()}} should also take {{List}} rather than
{{ArrayList}}.
# TestParallelImageWrite has 2 warnings (import and unused variable).
# The main question is how {{saveNamespace()}} works on a running NN.
The scenario here is that NN grabs {{FSNamesystem}} lock, then spawns several
threads,
and starts traversing the namespace tree and write inodes to the image file.
My understanding here is that the new threads are completely unaware of the
{{FSNamesystem}} synchronization or any other locks.
Could you please elaborate on this. I don't understand the intentions, but it
doesn't look right as is.
# It would be good to have a test case for the above condition. That is testing
saveNamespace() with multiple directories on a running mini-cluster.
# Last but not least, could you please share any performance measurements, if
you have any.
How much faster save goes if the directories are
#- on different drives,
#- on the same drive,
#- one on disk drive and another on NFS filer.
> savenamespace should write the fsimage to all configured fs.name.dir in
> parallel
> --------------------------------------------------------------------------------
>
> Key: HDFS-1071
> URL: https://issues.apache.org/jira/browse/HDFS-1071
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.patch
>
>
> If you have a large number of files in HDFS, the fsimage file is very big.
> When the namenode restarts, it writes a copy of the fsimage to all
> directories configured in fs.name.dir. This takes a long time, especially if
> there are many directories in fs.name.dir. Make the NN write the fsimage to
> all these directories in parallel.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
