[
https://issues.apache.org/jira/browse/HDFS-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202676#comment-13202676
]
Bikas Saha commented on HDFS-2910:
----------------------------------
Repro steps
1) Start 2 NN's in active standby mode
2) Remove write permissions from shared edits dir
3) Upon log roll triggered by standby, the active gets error when finalizing
the edit logs
4) FSEditLog.rollEditLogs() call endCurrentLogSegment() that hits exception.
This exception is silently ignored assuming logSync will fix it later on
5) FSEditLog.rollEditLogs() now calls startLogSegment() that again hits
exception. This exception is not silently ignored but is caught way up the
stack.
6) After that shared edits dir comes back online by giving write permissions
7) Standby triggers log roll again. But active NN keeps complaining about
BETWEEN_LOG_SEGMENTS with the following exception reported on the standby
12/02/07 00:46:55 INFO ha.EditLogTailer: Triggering log roll on remote NameNode
localhost/127.0.0.1:28000
12/02/07 00:47:53 WARN ha.EditLogTailer: Unable to trigger a roll of the active
NN
java.lang.IllegalStateException: java.lang.IllegalStateException: Bad state:
BETWEEN_LOG_SEGMENTS
at
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:887)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:831)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4026)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:741)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:116)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8068)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:439)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1608)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1604)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1602)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:203)
at $Proxy11.rollEditLog(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:162)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:256)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$3(EditLogTailer.java:253)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:287)
> Active NN reports Bad state: BETWEEN_LOG_SEGMENTS when shared edits dir is
> inaccessible during log roll
> -------------------------------------------------------------------------------------------------------
>
> Key: HDFS-2910
> URL: https://issues.apache.org/jira/browse/HDFS-2910
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
