[
https://issues.apache.org/jira/browse/AMQ-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damian Connolly updated AMQ-6090:
---------------------------------
Description:
Hi,
We are seeing a failure on the integrity of our LevelDB datastore which is
causing the broker to stop processing messages.
We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each
with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host. All
based on this guide: http://activemq.apache.org/replicated-leveldb-store.html
We have 10 Tomcat instances sending messages to the active broker, and 12
Tomcat instances reading from the VirtualQueue.
After several hours runtime, we see an exception in the log:
2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
{commandId = 7787648, responseRequired = false, messageId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13,
originalDestination = null, originalTransactionId = null, producerId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination =
topic://VirtualTopic.inventoryUsage, transactionId =
TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration =
0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190,
brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true,
type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId
= null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null,
dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
readOnlyProperties = false, readOnlyBody = false, droppable = false,
jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE"
value="27/1...ACHEREQUEST>} on:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped,
cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,195 | WARN | Async error occurred:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped |
org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException:
Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
java.io.IOException: Not running
The broker then stops and does not allow any messages to pass through until the
ActiveMQ node has been failed across manually.
I have attached my activemq.xml configuration.
Many thanks
was:
Hi,
We are seeing a failure on the integrity of our LevelDB datastore which is
causing the broker to stop processing messages.
We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each
with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host.
We have 10 Tomcat instances sending messages to the active broker, and 12
Tomcat instances reading from the VirtualQueue.
After several hours runtime, we see an exception in the log:
2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.IOException
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
{commandId = 7787648, responseRequired = false, messageId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13,
originalDestination = null, originalTransactionId = null, producerId =
ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination =
topic://VirtualTopic.inventoryUsage, transactionId =
TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration =
0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190,
brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true,
type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId
= null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null,
dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
readOnlyProperties = false, readOnlyBody = false, droppable = false,
jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE"
value="27/1...ACHEREQUEST>} on:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped,
cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,195 | WARN | Async error occurred:
org.apache.activemq.broker.SuppressReplyException: Store has been stopped |
org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
tcp:///10.241.163.60:40607@61616
2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException:
Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ
Transport: tcp:///10.241.163.60:40607@61616
java.io.IOException: Not running
The broker then stops and does not allow any messages to pass through until the
ActiveMQ node has been failed across manually.
I have attached my activemq.xml configuration.
Many thanks
> LevelDB IOException blocks broker
> ---------------------------------
>
> Key: AMQ-6090
> URL: https://issues.apache.org/jira/browse/AMQ-6090
> Project: ActiveMQ
> Issue Type: Bug
> Components: activemq-leveldb-store
> Affects Versions: 5.12.0
> Environment: Centos 6.5, Zookeeper 3.4.6, VMWare
> Reporter: Damian Connolly
> Attachments: activemq.xml, zoo.cfg
>
>
> Hi,
> We are seeing a failure on the integrity of our LevelDB datastore which is
> causing the broker to stop processing messages.
> We've set up ActiveMQ as a cluster of three nodes (master/slave/slave), each
> with a local Zookeeper node, each AMQ/ZK pair sitting on a VMWare host. All
> based on this guide: http://activemq.apache.org/replicated-leveldb-store.html
> We have 10 Tomcat instances sending messages to the active broker, and 12
> Tomcat instances reading from the VirtualQueue.
> After several hours runtime, we see an exception in the log:
> 2015-12-15 04:03:17,173 | INFO | Ignoring IO exception, java.io.IOException
> | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> handler.
> java.io.IOException
> at
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:552)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.LevelDBClient.store(LevelDBClient.scala:1390)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.apache.activemq.leveldb.DBManager$$anonfun$drainFlushes$1.apply$mcV$sp(DBManager.scala:627)[activemq-leveldb-store-5.12.0.jar:5.12.0]
> at
> org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)[hawtdispatch-scala-2.11-1.21.jar:1.21]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
> 2015-12-15 04:03:17,194 | INFO | Suppressing reply to: ActiveMQTextMessage
> {commandId = 7787648, responseRequired = false, messageId =
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1:13,
> originalDestination = null, originalTransactionId = null, producerId =
> ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:326123:1, destination
> = topic://VirtualTopic.inventoryUsage, transactionId =
> TX:ID:lonstct01asa.emea.kuoni.int-45593-1449862473131-1:4:963030, expiration
> = 0, timestamp = 1450152197189, arrival = 0, brokerInTime = 1450152197190,
> brokerOutTime = 0, correlationId = null, replyTo = null, persistent = true,
> type = null, priority = 4, groupID = null, groupSequence = 0,
> targetConsumerId = null, compressed = false, userID = null, content =
> org.apache.activemq.util.ByteSequence@68ecce89, marshalledProperties = null,
> dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
> readOnlyProperties = false, readOnlyBody = false, droppable = false,
> jmsXGroupFirstForConsumer = false, text = <CACHEREQUEST><ITEM ID="STAYDATE"
> value="27/1...ACHEREQUEST>} on:
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped,
> cause: null | org.apache.activemq.broker.TransportConnection | ActiveMQ
> Transport: tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,195 | WARN | Async error occurred:
> org.apache.activemq.broker.SuppressReplyException: Store has been stopped |
> org.apache.activemq.broker.TransportConnection.Service | ActiveMQ Transport:
> tcp:///10.241.163.60:40607@61616
> 2015-12-15 04:03:17,198 | INFO | Ignoring IO exception, java.io.IOException:
> Not running | org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ
> Transport: tcp:///10.241.163.60:40607@61616
> java.io.IOException: Not running
> The broker then stops and does not allow any messages to pass through until
> the ActiveMQ node has been failed across manually.
> I have attached my activemq.xml configuration.
> Many thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
