[
https://issues.apache.org/jira/browse/AMQ-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damian Connolly updated AMQ-6090:
---------------------------------
Attachment: zoo.cfg
activemq.xml
> LevelDB IOException blocks broker
> ---------------------------------
>
> Key: AMQ-6090
> URL: https://issues.apache.org/jira/browse/AMQ-6090
> Project: ActiveMQ
> Issue Type: Bug
> Components: activemq-leveldb-store
> Affects Versions: 5.12.0
> Environment: Centos 6.5, Zookeeper 3.4.6, VMWare
> Reporter: Damian Connolly
> Attachments: activemq.xml, zoo.cfg
>
>
> Hi,
> We are seeing a failure on the integrity of our LevelDB datastore which is
> causing the broker to stop processing messages.
> We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each
> with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host.
> We have 10 Tomcat instances sending messages to the active broker, and 12
> Tomcat instances reading from the VirtualQueue.
> After several hours runtime, we see an exception in the log:
> 2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException
> | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> handler.
> java.io.IOException
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
> 2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
> {commandId = 7787648, responseRequired = false, messageId =
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13,
> originalDestination = null, originalTransactionId = null, producerId =
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination
> = topic://VirtualTopic.inventoryUsage, transactionId =
> TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration
> = 0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190,
> brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true,
> type = null, priority = 4, groupID = null, groupSequence = 0,
> targetConsumerId = null, compressed = false, userID = null, content =
> org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null,
> dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
> readOnlyProperties = false, readOnlyBody = false, droppable = false,
> jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE"
> value="27/1...ACHEREQUEST>} on:
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped,
> cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ
> Transport: tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,195 | WARN | Async error occurred:
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped |
> org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
> tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException:
> Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ
> Transport: tcp:///10.241.163.60:40607@61616
> java.io.IOException: Not running
> The broker then stops and does not allow any messages to pass through until
> the ActiveMQ node has been failed across manually.
> I have attached my activemq.xml configuration.
> Many thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
