[
https://issues.apache.org/jira/browse/AMQ-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159187#comment-13159187
]
Timothy commented on AMQ-3293:
------------------------------
Hey Scott! I am facing the same problem, but the difference is that I just send
an byte message (around 100MB), and the "OutOfMemoryError" be thrown, the
following situation is just like you said. I was thinking is that really a bug?
or we have to tune the configuration for AMQ. Because if we really want to send
large messages, blob or stream message can be used. Did you try to send blob
message?
> ActiveMQ broker hangs after sending many large messages with a TTL, and no DLQ
> ------------------------------------------------------------------------------
>
> Key: AMQ-3293
> URL: https://issues.apache.org/jira/browse/AMQ-3293
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Linux 2.6.35-28-generic-pae #49-Ubuntu SMP Tue Mar 1
> 14:58:06 UTC 2011 i686 GNU/Linux
> java version "1.5.0_22"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
> Java HotSpot(TM) Server VM (build 1.5.0_22-b03, mixed mode)
> Reporter: Scott Emmons
> Priority: Critical
> Attachments: activemq.xml, jms.java
>
>
> While doing some ActiveMQ configuration testing I've run into some
> interesting where I can consistently make ActiveMQ 5.4.2 hang, unable even to
> shutdown. In this particular case, I have a producer that repeatedly sends
> persistent TextMessages containing random contents, of a random length up to
> 4MB to a single queue, with a 30 second TTL. You'll see in the configuration
> fragments below that we have sendFailIfNoSpace=true and are using
> vmQueueCursor with a DLQ policy of processExpired=false.
> At some point, the client will get a ResourceAllocationException, as expected
> once we hit the store limits. I'm running this client in a loop where it will
> keep trying, and as messages get expired, it will eventually start sending
> messages again. This will run for awhile, but then the producer will
> eventually hang. If I kill and restart the producer, it will continue to
> hang. The jetty console may show a couple of pending messages in the queue,
> but any consumer also hangs.
> Attempting to shut down activemq results in an error message that a
> particular connection "is taking a long time to shutdown", however there are
> no connections to the indicated port (as that producer was killed).
> You can see full thread dumps, lsof output, etc below.
> In another testcase, I was not using vmQueueCursor but this results in
> OutOfMemory exceptions repeatedly in ActiveMQ, and attempts to restart
> ActiveMQ hang right after kahadb journal recovery. I do not have a stacktrace
> or more info from this particular testcase.
> I can sanitize the producer client testcase if needed and attach it.
> ----------
> Full thread dump Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode):
> "Simple Discovery Agent:
> java.util.concurrent.ThreadPoolExecutor$Worker@520fa4" daemon prio=1
> tid=0x0a079150 nid=0x16c6 in Object.wait() [0x88d73000..0x88d74100]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90bc0818> (a java.lang.Object)
> at
> org.apache.activemq.transport.discovery.simple.SimpleDiscoveryAgent$1.run(SimpleDiscoveryAgent.java:136)
> - locked <0x90bc0818> (a java.lang.Object)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
> at java.lang.Thread.run(Thread.java:595)
> "NetworkBridge" daemon prio=1 tid=0x0a0819d8 nid=0x16c5 waiting on condition
> [0x883e0000..0x883e0d80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
> at
> java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Task" daemon prio=1 tid=0x0a337a10 nid=0x16be waiting on condition
> [0x87cfc000..0x87cfd100]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
> at
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:939)
> at java.lang.Thread.run(Thread.java:595)
> "Simple Discovery Agent:
> java.util.concurrent.ThreadPoolExecutor$Worker@1ab90e2" daemon prio=1
> tid=0x0a875038 nid=0x1621 in Object.wait() [0x891fc000..0x891fcf80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90bc0818> (a java.lang.Object)
> at
> org.apache.activemq.transport.discovery.simple.SimpleDiscoveryAgent$1.run(SimpleDiscoveryAgent.java:136)
> - locked <0x90bc0818> (a java.lang.Object)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Task" daemon prio=1 tid=0x09ba8158 nid=0x15ee waiting on condition
> [0x88059000..0x88059f80]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
> at
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:939)
> at java.lang.Thread.run(Thread.java:595)
> "Simple Discovery Agent:
> java.util.concurrent.ThreadPoolExecutor$Worker@12b6836" daemon prio=1
> tid=0x09c814a8 nid=0x15e0 waiting on condition [0x87c7b000..0x87c7bd80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
> at
> java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> "ConcurrentQueueStoreAndDispatch" daemon prio=1 tid=0x09ef2628 nid=0x157f
> waiting on condition [0x88ff8000..0x88ff8e00]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Transport: tcp:///10.4.64.66:34904" daemon prio=1 tid=0x09ba7ee8
> nid=0x1579 waiting on condition [0x88563000..0x88563e80]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
> at
> org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1599)
> at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:708)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:644)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:518)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:192)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:462)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:677)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)
> - locked <0xa08ab6d8> (a
> org.apache.activemq.transport.InactivityMonitor$1)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Connection Dispatcher: /10.4.64.66:34904" daemon prio=1
> tid=0x0a1a44d8 nid=0x1578 in Object.wait() [0x8b0fe000..0x8b0fef00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa08edc38> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0xa08edc38> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "ActiveMQ Connection Dispatcher: vm://cluster_a#6012" daemon prio=1
> tid=0x09fca838 nid=0x2c54 in Object.wait() [0x87fd8000..0x87fd9000]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa07b6438> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0xa07b6438> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "ActiveMQ Connection Dispatcher: vm://cluster_a#3116" daemon prio=1
> tid=0x0a7aeb38 nid=0x6139 in Object.wait() [0x87dfe000..0x87dfef80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa0791430> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0xa0791430> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "ActiveMQ Transport: tcp:///10.4.64.66:36292" daemon prio=1 tid=0x0a1a4ba0
> nid=0x1b15 waiting on condition [0x890fa000..0x890fb080]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1179)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:201)
> at java.util.concurrent.FutureTask.get(FutureTask.java:80)
> at
> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:718)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:644)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:518)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:192)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:462)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:677)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)
> - locked <0xa0013660> (a
> org.apache.activemq.transport.InactivityMonitor$1)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Connection Dispatcher: /10.4.64.66:36292" daemon prio=1
> tid=0x09caed28 nid=0x1b14 in Object.wait() [0x882de000..0x882df100]
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa00133c8> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0xa00133c8> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Topic ActiveMQ.Advisory.Consumer.Queue.orchestrationResponse" daemon prio=1
> tid=0x09e83730 nid=0xbad in Object.wait() [0x87d7d000..0x87d7e000]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x9167a250> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x9167a250> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Topic ActiveMQ.Advisory.Expired.Queue.orchestrationResponse" daemon prio=1
> tid=0x09e846b0 nid=0x17b in Object.wait() [0x884e2000..0x884e2d80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x916643d8> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x916643d8> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Topic ActiveMQ.Advisory.MessageDLQd.Queue.orchestrationResponse" daemon
> prio=1 tid=0x0a028150 nid=0x17a in Object.wait() [0x8835f000..0x8835fe00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x91662a10> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x91662a10> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Usage Async Task" daemon prio=1 tid=0x0a0c7e30 nid=0x7fdb waiting on
> condition [0x880da000..0x880dae80]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
> "Topic ActiveMQ.Advisory.Producer.Queue.orchestrationResponse" daemon prio=1
> tid=0x0a10d120 nid=0x7fd7 in Object.wait() [0x8815b000..0x8815bf00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90cc6118> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x90cc6118> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Topic ActiveMQ.Advisory.Queue" daemon prio=1 tid=0x0a844bb8 nid=0x7fd6 in
> Object.wait() [0x881dc000..0x881dcf80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90cc2f48> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x90cc2f48> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Queue:orchestrationResponse" daemon prio=1 tid=0x89680bc8 nid=0x7fd5 waiting
> on condition [0x8825d000..0x8825e000]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1179)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:201)
> at java.util.concurrent.FutureTask.get(FutureTask.java:80)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:337)
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:457)
> at
> org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:171)
> at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:797)
> at
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1527)
> at
> org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1519)
> at
> org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1584)
> at
> org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1574)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1686)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> - locked <0x90cb36c8> (a org.apache.activemq.broker.region.Queue$3)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Topic ActiveMQ.Advisory.Connection" daemon prio=1 tid=0x89783a98 nid=0x7fd4
> in Object.wait() [0x88461000..0x88462080]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90cc5188> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x90cc5188> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> "Timer-1" prio=1 tid=0x898f7bf8 nid=0x7f9c in Object.wait()
> [0x885e4000..0x885e4e80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90cf75c8> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x90cf75c8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
> "HashSessionScavenger-0" daemon prio=1 tid=0x898834c0 nid=0x7f9b in
> Object.wait() [0x88665000..0x88665f00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90d4a800> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x90d4a800> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
> "qtp14633980-49" prio=1 tid=0x895d3050 nid=0x7f9a waiting on condition
> [0x886e6000..0x886e6f80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-48" prio=1 tid=0x896688c0 nid=0x7f99 waiting on condition
> [0x88767000..0x88768000]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-47" prio=1 tid=0x896f5660 nid=0x7f98 waiting on condition
> [0x887e8000..0x887e9080]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-46" prio=1 tid=0x896f4940 nid=0x7f97 waiting on condition
> [0x88869000..0x8886a100]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-45" prio=1 tid=0x89552168 nid=0x7f96 waiting on condition
> [0x888ea000..0x888ead80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-44" prio=1 tid=0x89af3680 nid=0x7f95 waiting on condition
> [0x8896b000..0x8896be00]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-43" prio=1 tid=0x8afe1d38 nid=0x7f94 waiting on condition
> [0x88a6d000..0x88a6de80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
> at java.lang.Thread.run(Thread.java:595)
> "qtp14633980-42 - Acceptor0 [email protected]:7922" prio=1
> tid=0x89552d90 nid=0x7f93 runnable [0x889ec000..0x889ecf00]
> at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
> at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
> at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x90ca0a98> (a sun.nio.ch.Util$1)
> - locked <0x90ca0a88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x90ca09b0> (a sun.nio.ch.PollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:480)
> at
> org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
> at
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:134)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:793)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Transport Server: tcp://0.0.0.0:7921?transport.closeAsync=false"
> daemon prio=1 tid=0x898b32d0 nid=0x7f87 runnable [0x88c71000..0x88c72100]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> - locked <0x90bac140> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:280)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Transport Server Thread Handler:
> tcp://0.0.0.0:7921?transport.closeAsync=false" daemon prio=1 tid=0x8966b640
> nid=0x7f86 waiting on condition [0x88cf2000..0x88cf2d80]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:396)
> at
> org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:352)
> at java.lang.Thread.run(Thread.java:595)
> "ActiveMQ Data File Writer" daemon prio=1 tid=0x89a21840 nid=0x7f84 in
> Object.wait() [0x88df4000..0x88df4e80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90ba1318> (a
> org.apache.kahadb.journal.DataFileAppender$1)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:325)
> - locked <0x90ba1318> (a org.apache.kahadb.journal.DataFileAppender$1)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)
> "ActiveMQ Journal Checkpoint Worker" daemon prio=1 tid=0x898bf0b8 nid=0x7f83
> waiting on condition [0x88e75000..0x88e75f00]
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:286)
> "KahaDB Scheduler" daemon prio=1 tid=0x8c5d25c0 nid=0x7f82 in Object.wait()
> [0x88ef6000..0x88ef6f80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90ba0430> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x90ba0430> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
> "RMI LeaseChecker" daemon prio=1 tid=0x8afe0d38 nid=0x7f7e waiting on
> condition [0x89079000..0x89079d80]
> at java.lang.Thread.sleep(Native Method)
> at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
> at java.lang.Thread.run(Thread.java:595)
> "RMI RenewClean-[10.4.64.66:39231]" daemon prio=1 tid=0x0a337590 nid=0x7f7c
> in Object.wait() [0x8917b000..0x8917be80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90b9e178> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
> - locked <0x90b9e178> (a java.lang.ref.ReferenceQueue$Lock)
> at
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
> at java.lang.Thread.run(Thread.java:595)
> "GC Daemon" daemon prio=1 tid=0x0a295dc0 nid=0x7f7a in Object.wait()
> [0x8927d000..0x8927df80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90c48fe8> (a sun.misc.GC$LatencyLock)
> at sun.misc.GC$Daemon.run(GC.java:100)
> - locked <0x90c48fe8> (a sun.misc.GC$LatencyLock)
> "RMI Reaper" prio=1 tid=0x0a295620 nid=0x7f79 in Object.wait()
> [0x892fe000..0x892ff000]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90ae27e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
> - locked <0x90ae27e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
> at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
> at java.lang.Thread.run(Thread.java:595)
> "RMI TCP Accept-0" daemon prio=1 tid=0x0a295040 nid=0x7f78 runnable
> [0x894fe000..0x894ff080]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> - locked <0x90bc0380> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
> at java.lang.Thread.run(Thread.java:595)
> "RMI TCP Accept-7923" daemon prio=1 tid=0x898869b8 nid=0x7f76 runnable
> [0x8b2e2000..0x8b2e2d80]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> - locked <0x90b9e450> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
> at java.lang.Thread.run(Thread.java:595)
> "RMI TCP Accept-0" daemon prio=1 tid=0x8be438c8 nid=0x7f6d runnable
> [0x8bcb7000..0x8bcb7e80]
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> - locked <0x90b27a40> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:31)
> at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
> at java.lang.Thread.run(Thread.java:595)
> "Timer-0" daemon prio=1 tid=0x8be41cc8 nid=0x7f6c in Object.wait()
> [0x8bd38000..0x8bd38f00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90b27b28> (a java.util.TaskQueue)
> at java.lang.Object.wait(Object.java:474)
> at java.util.TimerThread.mainLoop(Timer.java:483)
> - locked <0x90b27b28> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
> "Low Memory Detector" daemon prio=1 tid=0x8c5cffe0 nid=0x7f6b runnable
> [0x00000000..0x00000000]
> "CompilerThread1" daemon prio=1 tid=0x8c5cec00 nid=0x7f6a waiting on
> condition [0x00000000..0x8c079228]
> "CompilerThread0" daemon prio=1 tid=0x8c5cdc80 nid=0x7f69 waiting on
> condition [0x00000000..0x8c0fa2a8]
> "AdapterThread" daemon prio=1 tid=0x8c5cccc0 nid=0x7f68 waiting on condition
> [0x00000000..0x00000000]
> "Signal Dispatcher" daemon prio=1 tid=0x8c5cbe88 nid=0x7f67 waiting on
> condition [0x00000000..0x00000000]
> "Finalizer" daemon prio=1 tid=0x8c5c2840 nid=0x7f66 in Object.wait()
> [0x8c47d000..0x8c47de00]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90b2a8e8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
> - locked <0x90b2a8e8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> "Reference Handler" daemon prio=1 tid=0x8c5c22f8 nid=0x7f65 in Object.wait()
> [0x8c4fe000..0x8c4fee80]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90b31778> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:474)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x90b31778> (a java.lang.ref.Reference$Lock)
> "main" prio=1 tid=0x09b460c0 nid=0x7f58 in Object.wait()
> [0xbff23000..0xbff24758]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x90d6bef0> (a [Z)
> at java.lang.Object.wait(Object.java:474)
> at
> org.apache.activemq.console.command.StartCommand.waitForShutdown(StartCommand.java:161)
> - locked <0x90d6bef0> (a [Z)
> at
> org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:104)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
> at
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:592)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
> at org.apache.activemq.console.Main.main(Main.java:107)
> "VM Thread" prio=1 tid=0x8c5bff38 nid=0x7f64 runnable
> "GC task thread#0 (ParallelGC)" prio=1 tid=0x09b60ad0 nid=0x7f60 runnable
> "GC task thread#1 (ParallelGC)" prio=1 tid=0x09b61720 nid=0x7f61 runnable
> "GC task thread#2 (ParallelGC)" prio=1 tid=0x09b62358 nid=0x7f62 runnable
> "GC task thread#3 (ParallelGC)" prio=1 tid=0x09b62f90 nid=0x7f63 runnable
> "VM Periodic Task Thread" prio=1 tid=0x8be4cb40 nid=0x7f6e waiting on
> condition
> ----------
> INFO | The connection to '/10.4.64.66:36292' is taking a long time to
> shutdown.
> INFO | The connection to '/10.4.64.66:36292' is taking a long time to
> shutdown.
> INFO | The connection to '/10.4.64.66:36292' is taking a long time to
> shutdown.
> INFO | The connection to '/10.4.64.66:36292' is taking a long time to
> shutdown.
> ----------
> java 32600 scott 6u IPv6 19919590 0t0 TCP *:59822 (LISTEN)
> java 32600 scott 9u IPv6 19919615 0t0 TCP *:7923 (LISTEN)
> java 32600 scott 12u IPv6 19919617 0t0 TCP *:39231 (LISTEN)
> java 32600 scott 14u IPv6 20371784 0t0 TCP
> h1:7922->h2:48935 (ESTABLISHED)
> java 32600 scott 18u IPv6 20371785 0t0 TCP
> h1:7922->h2:48936 (ESTABLISHED)
> java 32600 scott 20u IPv6 20371786 0t0 TCP
> h1:7922->h2:48937 (ESTABLISHED)
> java 32600 scott 24u IPv6 20371787 0t0 TCP
> h1:7922->h2:48938 (ESTABLISHED)
> java 32600 scott 29u IPv6 19919657 0t0 TCP *:7922 (LISTEN)
> java 32600 scott 30u IPv6 20371788 0t0 TCP
> h1:7922->h2:48939 (ESTABLISHED)
> java 32600 scott 31u IPv6 20371789 0t0 TCP
> h1:7922->h2:48940 (ESTABLISHED)
> ----------
> <destinationPolicy>
>
> <policyMap>
>
> <policyEntries>
>
> <!-- By default, do not use the DLQ, and use vmcursors -->
>
> <policyEntry queue=">">
>
> <deadLetterStrategy>
>
> <sharedDeadLetterStrategy processExpired="false"/>
>
> </deadLetterStrategy>
>
> <pendingQueuePolicy>
>
> <vmQueueCursor/>
>
> </pendingQueuePolicy>
>
> </policyEntry>
>
> </policyEntries>
>
> </policyMap>
>
> </destinationPolicy>
> [...]
> <systemUsage>
>
> <systemUsage sendFailIfNoSpace="true">
>
> <memoryUsage>
>
> <memoryUsage limit="256 mb"/>
>
> </memoryUsage>
>
> <storeUsage>
>
> <storeUsage limit="1 gb" name="foo"/>
>
> </storeUsage>
>
> <tempUsage>
>
> <tempUsage limit="100 mb"/>
>
> </tempUsage>
>
> </systemUsage>
>
> </systemUsage>
> ----------
> -rw-r--r-- 1 scott scott 36621377 2011-04-21 17:35 db-126.log
> -rw-r--r-- 1 scott scott 37281079 2011-04-21 17:35 db-127.log
> -rw-r--r-- 1 scott scott 33556041 2011-04-21 23:17 db-132.log
> -rw-r--r-- 1 scott scott 33030144 2011-04-22 09:25 db-134.log
> -rw-r--r-- 1 scott scott 53248 2011-04-22 09:25 db.data
> -rw-r--r-- 1 scott scott 32824 2011-04-22 09:25 db.redo
> -rw-r--r-- 1 scott scott 0 2011-04-21 17:17 lock
> du -sh on kahadb: 128M
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
