Ran failover test again and after about 25 or so fail overs I got the below
errors on the master. I did not see any other errors. I will try to run
with debug turned on next week to see it helps with debugging.
Note: even after this error i was able to do few more failover before things
stopped working properly.
After few more failovers, the system was failing over automatically
continuously within the cluster and producer and consumer stopped working.
ACTIVEMQ_DATA: /activemq_5_10/apache-activemq-5.10-SNAPSHOT/data
Loading message broker from: xbean:activemq.xml
java.io.IOException: invalid record position 1809753339 (file:
/solidstate/ldb/000000006a404a30.log, offset: 27155147)
at
org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:316)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$read$1.apply(RecordLog.scala:560)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:552)
at
org.apache.activemq.leveldb.RecordLog$$anonfun$get_reader$1.apply(RecordLog.scala:534)
at scala.Option.map(Option.scala:133)
at
org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:534)
at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:560)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply$mcV$sp(LevelDBClient.scala:734)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$replay_from$1.apply(LevelDBClient.scala:701)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:551)
at
org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:700)
at
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore$$anonfun$send_wal_ack$1.apply$mcV$sp(SlaveLevelDBStore.scala:175)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Restarting broker
Loading message broker from: xbean:activemq.xml
also see the followng during failover:
2013-11-22 11:55:32,192 | INFO | Stopping BrokerService[largeamq] due to
except ion, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler |
LevelDB IOException handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport
.java:39)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.sc
ala:554)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(Leve
lDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBCli
ent.scala:1320)
at
org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.s
cala:1244)
at
org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:
708)
at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverN
extMessages(LevelDBStore.scala:756)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBa
tch(QueueStorePrefetch.java:106)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBat
ch(AbstractStoreCursor.java:258)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext
(AbstractStoreCursor.java:145)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(St
oreQueueCursor.java:131)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.jav
a:1876)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:208
6)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(Pr
efetchSubscription.java:409)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(Abstract
Region.java:412)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBrok
er.java:457)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:82)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:82)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionB
roker.java:277)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBro
kerFilter.java:92)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(Tran
sportConnection.java:476)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:236)
at
org.apache.activemq.broker.TransportConnection.service(TransportConne
ction.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportC
onnection.java:149)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport
.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireForm
atNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(Abs
tractInactivityMonitor.java:270)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSup
port.java:83)
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTranspo
rt.java:138)
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTranspor
t.java:69)
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(Selector
Selection.java:94)
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker
.java:119)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.NullPointerException
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.appl
y(LevelDBClient.scala:1248)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.appl
y(LevelDBClient.scala:1244)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1322)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDBClien
t.scala:326)
at
org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(LevelDBC
lient.scala:328)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply$mcV$sp(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1
.apply(LevelDBClient.scala:1321)
at
org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.sc
ala:1015)
at
org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_i
ndex$1.apply(LevelDBClient.scala:1021)
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.sc
ala:551)
--
View this message in context:
http://activemq.2283324.n4.nabble.com/replicatedLevelDB-errors-after-failover-tp4674550p4674773.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
