[
https://issues.apache.org/jira/browse/HDFS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-5037:
------------------------------
Attachment: hdfs-5037-1.patch
Patch attached. This adds a background thread that wakes up every 5 minutes and
checks if the currently open edit log exceeds a # edits threshold (currently
set to 20 million). Both of the wakeup interval and threshold are configurable.
I also thought about putting a check into logEdits instead of having a
background thread, but it's a little messier and maybe more overhead on the
fast path. I can do that though if preferred.
> Active NN should trigger its own edit log rolls
> -----------------------------------------------
>
> Key: HDFS-5037
> URL: https://issues.apache.org/jira/browse/HDFS-5037
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, namenode
> Affects Versions: 3.0.0, 2.1.0-beta
> Reporter: Todd Lipcon
> Assignee: Andrew Wang
> Priority: Critical
> Attachments: hdfs-5037-1.patch
>
>
> We've seen cases where the SBN/2NN went down, and then users accumulated very
> very large edit log segments. This causes a slow startup time because the
> last edit log segment must be read fully to recover it before the NN can
> start up again. Additionally, in the case of QJM, it can trigger timeouts on
> recovery or edit log syncing because the very-large segment has to get
> processed within a certain time bound.
> We could easily improve this by having the NN trigger its own edit log rolls
> on a configurable size (eg every 256MB)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
