[
https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644134#action_12644134
]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
Most answers are in the document. I'll try to clarify.
# _Multiple standby nodes can be connected to the same active name-node._
Journal spool is an internal SNN mechanism that helps to delay applying journal
records to the SNN's memory namespace while SNN starts up or creates a
checkpoint. So if you have multiple standbys each of them will support its own
journal spool. Don't see a problem here.
The question is how the name-node handles rollEdits from the second SNN when
the checkpointing for the first one is still in progress. There are 2 ways to
handle this.
## Declare that NN can tolerate only one checkpoint at a time and refuse
checkpoint requests from other standbys until the current is complete. This is
the simplest approach, and I plan to implement it at least initially.
## NN can create multiple edits.new for each subsequent rollEdits request. That
is, when SNN1 asks NN to rollEdits NN creates edits.new.1
When SNN2 asks NN to rollEdits and edits.new.1 still exists NN creates
edits.new.2 and so on. This is more tricky but is doable too.
# _SNN treats each received journal record as a name-node command._
Journal records in hdfs are self-contained, that is they do not depend on the
current state of the name-server. Say, if a modification time should be changed
then the time to which it should be changed to is specified in the record, it
is not taken from SNN.system.currentTime().
# When SNN receives a record it applies it to the namespace in its memory AND
if any storage directories are specified for SNN it will write the record to
the edits file(s), just as NN would do, AND THEN will reply to the name-node
that operation is complete.
# Journal spool persistently stores records on a local disk.
# _NN should keep running even if the last storage directory fails. It should
enter safe mode in this case._
When all journal streams fail on the name-node it (NN) has no more way to
persist namespace modifications. Currently we treat that by shutting down the
name-node. Shutting down NN when the last hard drive fails does not work well
since we loose both the image file and the in-memory state. I think we should
keep NN running, but prevent it from name-space mutations. The safe-mode
prevents it from changes. Keeping it in running condition reserves a chance to
save the in-memory image to some other device manually mounted before calling
saveCheckpoint.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN
> so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
