[
https://issues.apache.org/jira/browse/AMQ-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199654#comment-13199654
]
James Furness commented on AMQ-3677:
------------------------------------
Stack trace from 5.5.1-fuse-01-20 (seems to happen only once this time rather
than every 30 seconds):
{code}
javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1869)
at
org.apache.activemq.ActiveMQConnection$2$1.run(ActiveMQConnection.java:1790)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null
at
org.apache.activemq.store.kahadb.MessageDatabase.removeAckLocation(MessageDatabase.java:1923)
at
org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1200)
at
org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:993)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
at
org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:991)
at
org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:926)
at
org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220)
at
org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:918)
at
org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:830)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.doAcknowledge(KahaDBStore.java:693)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.acknowledge(KahaDBStore.java:679)
at
org.apache.activemq.store.kahadb.KahaDBTransactionStore.acknowledge(KahaDBTransactionStore.java:492)
at
org.apache.activemq.store.kahadb.KahaDBTransactionStore$2.acknowledge(KahaDBTransactionStore.java:201)
at org.apache.activemq.broker.region.Topic.acknowledge(Topic.java:494)
at
org.apache.activemq.broker.region.DurableTopicSubscription.acknowledge(DurableTopicSubscription.java:270)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:229)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:427)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:560)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:276)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:87)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:460)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:229)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:291)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
... 3 common frames omitted
{code}
> NullPointerException in MessageDatabase leaves message stuck in KahaDB
> ----------------------------------------------------------------------
>
> Key: AMQ-3677
> URL: https://issues.apache.org/jira/browse/AMQ-3677
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.1
> Environment: ActiveMQ: 5.5.1-fuse-01-13
> JVM: 1.6.0_30-b12
> Reporter: James Furness
>
> A message appears to have gotten into a stuck state where it cannot be
> expired from KahaDB.
> Each expiry pass (every 30 seconds) is failing, presumably on the same
> message as the destination/expiration/timestamp/brokerInTime/etc are all
> identical:
> {code}
> 2012-01-23 14:02:48,191 ERROR ranteed.2] Scheduler BrokerLauncher
> org.apache.activemq.broker.ConnectionContext@7c0e6709:
> Persistent message has expired: ActiveMQBytesMessage {commandId = 1611,
> responseRequired = true, messageId =
> ID:broker1-59818-1327207516246-2:1:1:2:1, originalDestination = null,
> originalTransactionId = null, producerId =
> ID:broker-38443-1327100544854-21:1:1:1, destination = topic://TOPIC,
> transactionId = null, expiration = 1327294033981, timestamp = 1327207633981,
> arrival = 0, brokerInTime = 1327207633998, brokerOutTime = 1327207633993,
> correlationId = null, replyTo = null, persistent = true, type = null,
> priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null,
> compressed = false, userID = null, content =
> org.apache.activemq.util.ByteSequence@35fa419, marshalledProperties = null,
> dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
> readOnlyProperties = false, readOnlyBody = false, droppable = false}
> ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } for
> DurableTopicSubscription-NC_broker.1_inbound_broker.1.SystemGuaranteed.2:NC-DS_broker.1.SystemGuaranteed.2_TOPIC,
> id=OFFLINE:1:17, active=false, destinations=1, total=9, pending=9,
> dispatched=0, inflight=0, prefetchExtension=0
> 2012-01-23 14:02:48,191 ERROR ranteed.2] Scheduler BrokerLauncher
> Sent to DLQ: ActiveMQBytesMessage {commandId = 1611,
> responseRequired = true, messageId =
> ID:broker1-59818-1327207516246-2:1:1:2:1, originalDestination = null,
> originalTransactionId = null, producerId =
> ID:broker-38443-1327100544854-21:1:1:1, destination = topic://TOPIC,
> transactionId = null, expiration = 1327294033981, timestamp = 1327207633981,
> arrival = 0, brokerInTime = 1327207633998, brokerOutTime = 1327207633993,
> correlationId = null, replyTo = null, persistent = true, type = null,
> priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null,
> compressed = false, userID = null, content =
> org.apache.activemq.util.ByteSequence@35fa419, marshalledProperties = null,
> dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
> readOnlyProperties = false, readOnlyBody = false, droppable = false}
> ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null }
> DurableTopicSubscription-NC_broker.1_inbound_broker.2:NC-DS_broker.2_TOPIC,
> id=OFFLINE:1:17, active=false, destinations=1, total=9, pending=9,
> dispatched=0, inflight=0, prefetchExtension=0
> org.apache.activemq.broker.ConnectionContext@7c0e6709
> 2012-01-23 14:02:48,192 ERROR ranteed.2] Scheduler o.a.a.broker.region.Topic
> Failed to remove expired Message from the store
> java.lang.NullPointerException: null
> at
> org.apache.activemq.store.kahadb.MessageDatabase.removeAckLocation(MessageDatabase.java:1920)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1197)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:990)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
> ~[kahadb-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:988)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:923)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:915)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:827)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.doAcknowledge(KahaDBStore.java:693)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.acknowledge(KahaDBStore.java:679)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore.acknowledge(KahaDBTransactionStore.java:492)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore$2.acknowledge(KahaDBTransactionStore.java:201)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.Topic.acknowledge(Topic.java:494)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.Topic.messageExpired(Topic.java:686)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.activemq.broker.region.Topic.doBrowse(Topic.java:568)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.activemq.broker.region.Topic.access$100(Topic.java:63)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.activemq.broker.region.Topic$6.run(Topic.java:667)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at java.util.TimerThread.mainLoop(Unknown Source) [na:1.6.0_30]
> at java.util.TimerThread.run(Unknown Source) [na:1.6.0_30]
> {code}
> Restarting the broker does not help. Restarting the client causes the same
> exception to be serialised to the client:
> {code}
> 2012-01-23 01:40:49,903 ERROR true&soTimeout=60000 ErrorListener
> Error
> javax.jms.JMSException: java.lang.NullPointerException
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1855)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.ActiveMQConnection$2$1.run(ActiveMQConnection.java:1776)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source) [na:1.6.0_30]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.6.0_30]
> at java.lang.Thread.run(Unknown Source) [na:1.6.0_30]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.activemq.store.kahadb.MessageDatabase.removeAckLocation(MessageDatabase.java:1920)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1197)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:990)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
> ~[kahadb-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:988)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:923)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:915)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:827)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.doAcknowledge(KahaDBStore.java:693)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.acknowledge(KahaDBStore.java:679)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore.acknowledge(KahaDBTransactionStore.java:492)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore$2.acknowledge(KahaDBTransactionStore.java:201)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.Topic.acknowledge(Topic.java:494)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.DurableTopicSubscription.acknowledge(DurableTopicSubscription.java:270)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:229)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:427)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:568)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:276)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:87)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:87)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:477)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at org.apache.activemq.command.MessageAck.visit(MessageAck.java:229)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:318)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> at
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
> ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
> ... 3 common frames omitted
> {code}
> Is there any mechanism to purge the stuck message (without blowing away the
> whole database), or deal with the situation automatically?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
