[
https://issues.apache.org/jira/browse/HDFS-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2824:
---------------------------------
Attachment: HDFS-2824-HDFS-1623.patch
Thanks a lot for the review, Todd. Here's an updated patch which addresses your
comments.
The issue here is that the upon starting a new edit log file, we first create a
new file, then write the header and pre-allocation bytes, before ever writing
the START_LOG_SEGMENT transaction. Thus, having zero-length or zero
transactions are perfectly valid states to find an edit log file, but these
violate one of the invariants introduced by HDFS-1073, that a non-empty edit
log file should always contain at least the START_LOG_SEGMENT transaction.
Furthermore, some of the sanity checks and tests conclude that the presence of
an edit log file with some txid X in the name therefore necessitates that txid
X has actually been written to disk, which is not necessarily the case.
To address this, we conclude that:
1. If we ever encounter a truly empty edits_inprogress file, just remove it.
2. If we ever encounter an edits_inprogress file with a valid header but no
transactions, just remove it.
3. If we ever encounter an invalid header in an edits_inprogress file, bail
out. This could be a legitimate corruption, or a mismatch of layoutVersions.
4. We can change FileJournalManager to make findMaxTransaction stateless.
5. The test case introduced in HDFS-2093 is now wrong, since it verifies that
the NN won't start if there's an empty edits log file at some txid X.
> HA: failover does not succeed if prior NN died just after creating an edit
> log segment
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2824
> URL: https://issues.apache.org/jira/browse/HDFS-2824
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: HDFS-2824-HDFS-1623.patch, HDFS-2824-HDFS-1623.patch
>
>
> In stress testing failover, I had the following failure:
> - NN1 rolls edit logs and starts writing edits_inprogress_1000
> - NN1 crashes before writing the START_LOG_SEGMENT transaction
> - NN2 tries to become active, and calls {{recoverUnfinalizedSegment}}. Since
> the log file contains no valid transactions, it is marked as corrupt and
> renamed with the {{.corrupt}} suffix
> - The sanity check in {{openLogsForWrite}} will refuse to open a new
> in-progress log at the same txid. Failover does not proceed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
