Your subject implies that you're unable to send (which I read to mean "unable to send any messages, at all, to this topic"), but looking at the thread dump, I see the nine threads in BLOCKED state waiting to acquire the same lock and a tenth thread that's holding that lock while doing work (see thread dump below). That tenth thread doesn't show any signs of being unable to make forward progress, which implies that it will eventually release the lock, and one of the other threads will then be able to make progress. If that's accurate, then there would be thread contention and slow performance, but publishers would still be able to make (slow) progress on sending messages. Is that what you're in fact seeing?
What would give us more information would be for you to take a CPU sampling snapshot from JVisualVM while you're in this state and share it here, to help us figure out where the time is being spent by the thread that's holding the lock. Operations done while holding locks should always be as fast as possible, but maybe something in your specific use case is causing the operations that occur while holding the lock for that topic to be slower than expected, and the CPU sampling snapshot would let one of us dig into whether that's the case and what it is (and what might be the root cause of it). And what more can you tell us about the load that's being placed on the broker and the patterns of publication and consumption: the number of consumers on the topic in question, the rate of message publication to the topic, whether selectors are in use on this topic's consumers, etc.? Is this thread dump captured during a load test (where you're explicitly trying to drive your system faster than it would normally go) or during normal operations? Thanks, Tim On Fri, Oct 25, 2019, 5:23 PM cart_chiu <[email protected]> wrote: > hi, we are experiencing an issue in our production environment when we > publish about 10k small messages. I see 9 BLOCKED threads in the thread > dump: > > priority:10 - threadId:0x00007f34b0070800 - nativeId:0x19c20 - nativeId > (decimal):105504 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:423) > - waiting to lock <0x000000076c328620> (a > org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:404) > at > > org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365) > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:523) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > at > > org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) > at > > org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) > at > > org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) > at > > org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:458) > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) > at > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306) > at > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) > at > > org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) > at > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > at > > org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227) > - locked <0x00000007214c0ec8> (a > org.apache.activemq.transport.InactivityMonitor$1) > at > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220) > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202) > at java.lang.Thread.run(Thread.java:745) > Locked ownable synchronizers: > - None > > Any ideas how we can tune our ActiveMQ settings to address this? > NabbleThreadDump.txt > <http://activemq.2283324.n4.nabble.com/file/t379693/NabbleThreadDump.txt> > > NabbleActiveMQ.txt > <http://activemq.2283324.n4.nabble.com/file/t379693/NabbleActiveMQ.txt> > Attached is our full thread dump and activemq.xml. Thanks in advance! > > > > -- > Sent from: > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >
