[
https://issues.apache.org/jira/browse/ZOOKEEPER-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573389#comment-14573389
]
Hongchao Deng commented on ZOOKEEPER-1546:
------------------------------------------
Is this JIRA related to ZOOKEEPER-1573? I think they are duplicate.
> "Unable to load database on disk" when restarting after node freeze
> -------------------------------------------------------------------
>
> Key: ZOOKEEPER-1546
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1546
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.3.5
> Reporter: Erik Forsberg
>
> One of my zookeeper servers in a quorum of 3 froze (probably due to
> underlying hardware problems). When restarting, zookeeper fails to start with
> the following in zookeeper.log:
> {noformat}
> 2012-09-04 09:02:35,300 - INFO [main:QuorumPeerConfig@90] - Reading
> configuration from: /etc/zookeeper/zoo.cfg
> 2012-09-04 09:02:35,316 - INFO [main:QuorumPeerConfig@310] - Defaulting to
> majority quorums
> 2012-09-04 09:02:35,333 - INFO [main:QuorumPeerMain@119] - Starting quorum
> peer
> 2012-09-04 09:02:35,358 - INFO [main:NIOServerCnxn$Factory@143] - binding to
> port 0.0.0.0/0.0.0.0:2181
> 2012-09-04 09:02:35,379 - INFO [main:QuorumPeer@819] - tickTime set to 2000
> 2012-09-04 09:02:35,380 - INFO [main:QuorumPeer@830] - minSessionTimeout set
> to -1
> 2012-09-04 09:02:35,380 - INFO [main:QuorumPeer@841] - maxSessionTimeout set
> to -1
> 2012-09-04 09:02:35,386 - INFO [main:QuorumPeer@856] - initLimit set to 10
> 2012-09-04 09:02:35,523 - INFO [main:FileSnap@82] - Reading snapshot
> /var/zookeeper/version-2/snapshot.500017240
> 2012-09-04 09:02:38,944 - ERROR [main:FileTxnSnapLog@226] - Failed to
> increment parent cversion for:
> /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
> at
> org.apache.zookeeper.server.DataTree.incrementCversion(DataTree.java:1218)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:224)
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:152)
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
> 2012-09-04 09:02:38,945 - FATAL [main:QuorumPeer@400] - Unable to load
> database on disk
> java.io.IOException: Failed to process transaction type: 2 error:
> KeeperErrorCode = NoNode for
> /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:154)
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
> 2012-09-04 09:02:38,946 - FATAL [main:QuorumPeerMain@87] - Unexpected
> exception, exiting abnormally
> java.lang.RuntimeException: Unable to run quorum server
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:401)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:143)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:103)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
> Caused by: java.io.IOException: Failed to process transaction type: 2 error:
> KeeperErrorCode = NoNode for
> /osp/production/scheduler/waitfordeps_tasks/per_period-3092724ef4d611e18411525400fff018-bulkload_histograms
> at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:154)
> at
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:222)
> at
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:398)
> ... 3 more
> {noformat}
> Removing data from /var/zookeeper/version-2 then restart seems to "fix" the
> problem (it gets a snapshot from one of the other nodes in the quorum).
> This is Zookeeper 3.3.5+19.5-1~squeeze-cdh3, i.e. from Cloudera's
> distribution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
