[
https://issues.apache.org/jira/browse/HADOOP-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463116
]
dhruba borthakur commented on HADOOP-227:
-----------------------------------------
My theory is that a checkpoint time is actually dominated by the time to read
and write the image to disk and the time to transfer the the image back and
forth between the namenode and the secondary namenode. The actual CPU time
needed to do the merge of the edit log with the fsimage may be relatively small.
My back-of-the-envelope calculation shows that sequential disk IO rates are at
best 2GB per minute. Assuming a 2 GB image:
1. NameNode reads image = 1min
2. Secondary NameNode stores image = 1 min (maybe in parallel with Step 1)
3. Secondary NameNode stores new image = 1min
4. NameNode stores image = 1 min
Thus a total of 3 to 4 minutes can be used to do a checkpoint (ignoring network
issues).
> Namespace check pointing is not performed until the namenode restarts.
> ----------------------------------------------------------------------
>
> Key: HADOOP-227
> URL: https://issues.apache.org/jira/browse/HADOOP-227
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.2.0
> Reporter: Konstantin Shvachko
> Assigned To: dhruba borthakur
> Attachments: patch-async-checkpoints-0.9.0,
> patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0,
> periodiccheckpoint.patch, periodiccheckpoint2.patch
>
>
> In current implementation when the name node starts, it reads its image file,
> then
> the edits file, and then saves the updated image back into the image file.
> The image file is never updated after that.
> In order to provide the system reliability reliability the namespace
> information should
> be check pointed periodically, and the edits file should be kept relatively
> small.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
