[ 
https://issues.apache.org/jira/browse/AMQ-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503455#comment-15503455
 ] 

William Crowell commented on AMQ-6090:
--------------------------------------

Did anyone find out what the result of this was?  I am seeing the same issue 
with 5.13.3 using LevelDB.

> LevelDB IOException blocks broker
> ---------------------------------
>
>                 Key: AMQ-6090
>                 URL: https://issues.apache.org/jira/browse/AMQ-6090
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-leveldb-store
>    Affects Versions: 5.12.0
>         Environment: Centos 6.5, Zookeeper 3.4.6, VMWare
>            Reporter: Damian Connolly
>         Attachments: activemq.xml, zoo.cfg
>
>
> Hi, 
> We are seeing a failure on the integrity of our LevelDB datastore which is 
> causing the broker to stop processing messages.
> We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each 
> with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host. All 
> based on this guide: http://activemq.apache.org/replicated-leveldb-store.html
> We have 10 Tomcat instances sending messages to the active broker, and 12 
> Tomcat instances reading from the VirtualQueue.
> After several hours runtime, we see an exception in the log:
> 2015-12-15 04:03:17,173 | INFO  | Ignoring IO exception, java.io.IOException 
> | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException 
> handler.
> java.io.IOException
>         at 
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
>         at 
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
>         at 
> org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
>         at 
> org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
>         at 
> org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
>         at 
> org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
>         at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
> 2015-12-15 04:03:17,194 | INFO  | Suppressing reply to: ActiveMQTextMessage 
> {commandId = 7787648, responseRequired = false, messageId = 
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13, 
> originalDestination = null, originalTransactionId = null, producerId = 
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination 
> = topic://VirtualTopic.inventoryUsage, transactionId = 
> TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration 
> = 0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190, 
> brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true, 
> type = null, priority = 4, groupID = null, groupSequence = 0, 
> targetConsumerId = null, compressed = false, userID = null, content = 
> org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null, 
> dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, 
> readOnlyProperties = false, readOnlyBody = false, droppable = false, 
> jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE" 
> value="27/1...ACHEREQUEST>} on: 
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped, 
> cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ 
> Transport: tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,195 | WARN  | Async error occurred: 
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped | 
> org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport: 
> tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,198 | INFO  | Ignoring IO exception, java.io.IOException: 
> Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ 
> Transport: tcp:///10.241.163.60:40607@61616
> java.io.IOException: Not running
> The broker then stops and does not allow any messages to pass through until 
> the ActiveMQ node has been failed across manually.
> I have attached my activemq.xml configuration.
> Many thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to