[
https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853441#action_12853441
]
Suresh Srinivas commented on HDFS-1073:
---------------------------------------
> The pro of txid numbering is that we can detect the case where some middle
> log got truncated. For example, if we have edits_0, edits_1000, and
> edits_2000, but edits_1000 only contains 500 edits, we can fail at that point
We should generate txid serially. With that it would be easy to determine the
missing logs, based on gaps in txid, even if we decide on naming the files
without txids.
> "it decouples the split (ie roll) of the edit log and the checkpoint of the
> image"
Currently a new edits are rolled when backup node asks the namenode to create
one. NN could have a policy where for every N number of transaction, it could
choose to roll edits. Backup NN could choose to checkpoint fsimage with the
relevant edits that have been rolled so far. This makes rolling of edits a
local NN policy and decouples it from backup NN.
> There may be occasional times when we want to roll the edit log when we've
> not made any edits into it...
To avoid this kind of complications - if NN periodically rolls edits,
checkpointing could be merging fsimage with all the rolled edits so far. In
such a case we need distinction between edits that are finalized and edit that
is being written to.
> Simpler model for Namenode's fs Image and edit Logs
> ----------------------------------------------------
>
> Key: HDFS-1073
> URL: https://issues.apache.org/jira/browse/HDFS-1073
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Sanjay Radia
> Assignee: Todd Lipcon
>
> The naming and handling of NN's fsImage and edit logs can be significantly
> improved resulting simpler and more robust code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
