Joseph Chang created AMQ-4239:
---------------------------------
Summary: Possible Race Condition in FilePendingMessageCursor when
sending messages quickly
Key: AMQ-4239
URL: https://issues.apache.org/jira/browse/AMQ-4239
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.7.0
Environment: OS : CentOS release 6.3 (Final)
KERNEL : 2.6.32-279.el6.x86_64
CPU : Intel(R) Xeon(R) CPU X3470 @ 2.93GHz
CPU COUNT : 8
CPU TYPE : x86_64
RAM : 32 GB
Manufacturer : Dell Inc.
Model : PowerEdge R310
BIOS Version : 1.8.2
Reporter: Joseph Chang
When sending persistent messages extremely quickly, we often see the following
stack trace.
Broker Persistence is turned on, but not using KahaDB. Instead, using JDBC
Persistence (MySQL). KahaDB is commented out on the xml configuration file.
Reading through source, seems like the Kaha engine is still used for temporary
storage of pending messages (/data/**/tmp_storage). However, in reading
through code, seems like the "loaded" member variable is not intialized in
time, causing this stack to be thrown.
Any help appreciated, simply gums up our logging, and the exceptions make it
back to the client.
[04 Dec 2012 08:02:08] [scheduler_Worker-2] 36391186 ERROR
(com.ms.jms.publisher.BaseMessagePublisher:143) -
java.lang.IllegalStateException: TheListIndex is not loaded
javax.jms.JMSException: java.lang.IllegalStateException: TheListIndex
is not loaded
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1305)
at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1795)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:269)
at
org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:121)
at
com.ms.jms.publisher.BaseMessagePublisher.talker(BaseMessagePublisher.java:133)
at
com.ms.jms.publisher.BaseMessagePublisher.sendMessage(BaseMessagePublisher.java:63)
at
com.ms.mscm.bus.ClientBasedMessagePublisher.sendMessage(ClientBasedMessagePublisher.java:169)
at
com.ms.cart.ScheduledCartProcessor.checkout(ScheduledCartProcessor.java:101)
at
com.ms.cart.ScheduledCartJob.doExecuteInternal(ScheduledCartJob.java:63)
at
com.ms.cart.ScheduledCartJob.executeInternal(ScheduledCartJob.java:50)
at
org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: java.lang.RuntimeException:
java.lang.IllegalStateException: TheListIndex is not loaded
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:239)
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:202)
at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1669)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:785)
at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:680)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:152)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:256)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalStateException: TheListIndex is not loaded
at org.apache.kahadb.index.ListIndex.assertLoaded(ListIndex.java:301)
at org.apache.kahadb.index.ListIndex.add(ListIndex.java:199)
at org.apache.activemq.store.kahadb.plist.PList$2.execute(PList.java:84)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
at org.apache.activemq.store.kahadb.plist.PList.addLast(PList.java:82)
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:438)
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:226)
... 27 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
