[
https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233099#comment-13233099
]
Todd Lipcon commented on HDFS-3004:
-----------------------------------
The EditLogInputExceptions are currently being thrown by this code:
{code}
try {
if ((op = in.readOp()) == null) {
break;
}
} catch (IOException ioe) {
long badTxId = txId + 1; // because txId hasn't been incremented yet
String errorMessage = formatEditLogReplayError(in,
recentOpcodeOffsets, badTxId);
FSImage.LOG.error(errorMessage);
throw new EditLogInputException(errorMessage,
ioe, numEdits);
}
{code}
It indicates that whatever exception happened was due to a deserialization
error, which is distinct from an application error.
EditLogTailer is used by the HA StandbyNode to tail the edits out of the edit
log and apply them to the SBN's namespace. Since it's reading the same log that
the active is writing, it's possible that it can see a partial edit at the end
of the file, in which case it will generally see an IOException. The fact that
it's being wrapped with EditLogInputException indicates that it was some
problem reading the edits and can likely be retried. If the EditLogTailer gets
a different type of exception, though, indicating that the _appplication_ of
the edit failed, then it will exit, because it may have left the namespace in
an inconsistent state and thus is no longer a candidate for failover.
> Implement Recovery Mode
> -----------------------
>
> Key: HDFS-3004
> URL: https://issues.apache.org/jira/browse/HDFS-3004
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: tools
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch,
> HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch,
> HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch,
> HDFS-3004.019.patch, HDFS-3004__namenode_recovery_tool.txt
>
>
> When the NameNode metadata is corrupt for some reason, we want to be able to
> fix it. Obviously, we would prefer never to get in this case. In a perfect
> world, we never would. However, bad data on disk can happen from time to
> time, because of hardware errors or misconfigurations. In the past we have
> had to correct it manually, which is time-consuming and which can result in
> downtime.
> Recovery mode is initialized by the system administrator. When the NameNode
> starts up in Recovery Mode, it will try to load the FSImage file, apply all
> the edits from the edits log, and then write out a new image. Then it will
> shut down.
> Unlike in the normal startup process, the recovery mode startup process will
> be interactive. When the NameNode finds something that is inconsistent, it
> will prompt the operator as to what it should do. The operator can also
> choose to take the first option for all prompts by starting up with the '-f'
> flag, or typing 'a' at one of the prompts.
> I have reused as much code as possible from the NameNode in this tool.
> Hopefully, the effort that was spent developing this will also make the
> NameNode editLog and image processing even more robust than it already is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
