Damian Connolly created AMQ-6090:
------------------------------------
Summary: LevelDB IOException blocks broker
Key: AMQ-6090
URL: https://issues.apache.org/jira/browse/AMQ-6090
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.12.0
Environment: Centos 6.5, Zookeeper 3.4.6, VMWare
Reporter: Damian Connolly
Hi,
We are seeing a failure on the integrity of our LevelDB datastore which is
causing the broker to stop processing messages.
We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each
with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host.
We have 10 Tomcat instances sending messages to the active broker, and 12
Tomcat instances reading from the VirtualQueue.
After several hours runtime, we see an exception in the log:
2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
{commandId = 7787648, responseRequired = false, messageId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13,
originalDestination = null, originalTransactionId = null, producerId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination =
topic://VirtualTopic.inventoryUsage, transactionId =
TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration =
0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190,
brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true,
type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId
= null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null,
dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
readOnlyProperties = false, readOnlyBody = false, droppable = false,
jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE"
value="27/1...ACHEREQUEST>} on:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped,
cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,195 | WARN | Async error occurred:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped |
org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException:
Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
java.io.IOException: Not running
The broker then stops and does not allow any messages to pass through until the
ActiveMQ node has been failed across manually.
I have attached my activemq.xml configuration.
Many thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
