Active Namenode shutdown after throwing java.io.IOException: BookKeeper error
during close error.
-------------------------------------------------------------------------------------------------
Key: BOOKKEEPER-219
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-219
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-client
Affects Versions: 4.0.0
Environment: 10.18.40.45: Active namenode
10.18.40.91: Backup namenode
10.18.52.157 : one zookeeper, three bookkeeper
Reporter: surendra singh lilhore
scenario:
1. I am started Active namenode and backup namenode.
2. write some file.
Result:
ANN throwing following Error
{noformat}
2012-04-18 10:46:04,001 WARN org.apache.bookkeeper.client.LedgerHandle:
Conditional write failed: BADVERSION
2012-04-18 10:46:04,001 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Error: finalize log segment 1, 8 failed for required journal
(JournalAndStream(mgr=org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager@1ec3362f,
stream=org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream@221a5770))
java.io.IOException: BookKeeper error during close
at
org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream.close(BookKeeperEditLogOutputStream.java:90)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:79)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:180)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:322)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:925)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:855)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:971)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4092)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:714)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:113)
at
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8068)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:417)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:891)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1661)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1657)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1205)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1655)
Caused by: org.apache.bookkeeper.client.BKException$ZKException
at org.apache.bookkeeper.client.BKException.create(BKException.java:64)
at
org.apache.bookkeeper.client.LedgerHandle.close(LedgerHandle.java:216)
at
org.apache.hadoop.contrib.bkjournal.BookKeeperEditLogOutputStream.close(BookKeeperEditLogOutputStream.java:86)
... 19 more
{noformat}
BNN throwing following Error
{noformat}
2012-04-18 10:39:09,421 ERROR org.apache.bookkeeper.client.PendingReadOp:
Error: No such entry while reading entry: 1 ledgerId: 5 from bookie:
/10.18.52.157:3183
2012-04-18 10:39:09,423 ERROR org.apache.bookkeeper.client.PendingReadOp:
Error: No such entry while reading entry: 1 ledgerId: 5 from bookie:
/10.18.52.157:3181
2012-04-18 10:39:09,457 WARN
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Edits tailer failed to
find any streams. Will try again later.
java.io.IOException: No ledger for fromTxnId -12344 found.
at
org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.getInputStream(BookKeeperJournalManager.java:329)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:246)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1100)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:206)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$700(EditLogTailer.java:59)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:318)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:276)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:293)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:504)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:289)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
