[
https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440421#comment-15440421
]
Hanisha Koneru edited comment on HDFS-4025 at 8/27/16 1:41 AM:
---------------------------------------------------------------
The 000.patch would start a syncJournal thread in each JournalNode. This thread
periodically syncs the journal node by getting the edit log manifest from
another randomly picked journal node. If the other journal node has an edit log
which is missing from the current journal node, it is downloaded from the other
journal node's http server. The in progress log segments are not synced in this
process.
Thank you [~jingzhao] for all the guidance.
was (Author: hanishakoneru):
This patch would start a syncJournal thread in each JournalNode. This thread
periodically syncs the journal node by getting the edit log manifest from
another randomly picked journal node. If the other journal node has an edit log
which is missing from the current journal node, it is downloaded from the other
journal node's http server. The in progress log segments are not synced in this
process.
Thank you [~jingzhao] for all the guidance.
> QJM: Sychronize past log segments to JNs that missed them
> ---------------------------------------------------------
>
> Key: HDFS-4025
> URL: https://issues.apache.org/jira/browse/HDFS-4025
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: QuorumJournalManager (HDFS-3077)
>
> Attachments: HDFS-4025.000.patch
>
>
> Currently, if a JournalManager crashes and misses some segment of logs, and
> then comes back, it will be re-added as a valid part of the quorum on the
> next log roll. However, it will not have a complete history of log segments
> (i.e any individual JN may have gaps in its transaction history). This
> mirrors the behavior of the NameNode when there are multiple local
> directories specified.
> However, it would be better if a background thread noticed these gaps and
> "filled them in" by grabbing the segments from other JournalNodes. This
> increases the resilience of the system when JournalNodes get reformatted or
> otherwise lose their local disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
