[
https://issues.apache.org/jira/browse/HDFS-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449123#comment-13449123
]
Todd Lipcon commented on HDFS-3867:
-----------------------------------
There's actually an *Option 3* as well:
To support the rolling restart use case, we can simply have the rolling restart
script perform the following sequence:
{code}
for jn in $NODES ; do
ssh $jn service hadoop-journalnode restart
sleep 5 # give time for JN to come back up -- could instead poll its web UI
until it's back up
hdfs dfsadmin -rollEdits
# SANITY: verify through metrics that the NN is now writing to the restarted
JN
done
{code}
The manual rollEdits command is enough to pick up the now-restarted JN, so this
process provides a robust rolling restart. So long as this is well documented,
I think it is a sufficient solution for now. A careful administrator will want
to perform the "SANITY" step between each rolled daemon regardless, so asking
them to also manually re-incorporate the restarted daemon doesn't seem too
onerous.
I'm continuing to work on the implementation of automatically triggered log
rolls, since it's useful even outside the context of a rolling restart. For
example, if one of the JNs has a network blip and is down for a few minutes, it
would be nice to automatically re-add it when it comes back up, instead of
waiting for a log roll. But, this is a more general improvement that also
affects file-based edits storage: currently we do not re-add an NFS mounted
edits directory after such a blip until the next roll either.
Given this, I'd like to consider moving this out of the HDFS-3077 branch and
rephrasing the feature more generally as:
{quote}
Currently, when journal managers have failed, they are only re-added on the
next roll of the edit log. It would be better to periodically monitor the state
of the journal manager, and, when the journal manager sees that its storage is
accessible again, automatically trigger a log roll so that it can be activated
again. In the context of file journal managers, this would be useful to
automatically restore writing to an NFS filer after a network blip. In the case
of the QuorumJournalManager, it would be useful to automatically restore full
fault tolerance after a JN has suffered a lapse in availability.
{quote}
Thoughts?
> QJM: Support rolling restart of JNs
> -----------------------------------
>
> Key: HDFS-3867
> URL: https://issues.apache.org/jira/browse/HDFS-3867
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> In order to perform upgrades or other maintenance, it is useful to be able to
> perform a rolling restart of the journal nodes while the NameNode is active.
> Currently, this does not work, because the NN only picks up restarted JNs
> again on the beginning of the next log segment. So, if the NN does not roll
> after each node is restarted in turn, the NN will eventually fail to commit
> to a quorum and crash.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
