[
https://issues.apache.org/jira/browse/AMQ-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dejan Bosanac updated AMQ-2749:
-------------------------------
Fix Version/s: (was: 5.4.2)
5.5.0
> Broker hangs in TopicSubscription.add
> -------------------------------------
>
> Key: AMQ-2749
> URL: https://issues.apache.org/jira/browse/AMQ-2749
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Environment: red had enterprise
> Reporter: Dennis Klinkott
> Fix For: 5.5.0
>
> Attachments: activemq.xml, jmx console screenshot.jpg,
> jstack.2010-05-25-14-01-30, jstack.2010-05-25-14-10-54
>
>
> Our broker suddenly hang in production environment. We created a bunch of
> jstacks to see what the reason is.
> The jstacks show that the broker waits with many threads in doMessageSend:
> e.g.
> "ActiveMQ Transport: tcp:///10.231.233.117:51088" daemon prio=10
> tid=0x00002aab84016800 nid=0x1f19 waiting for monitor entry
> [0x000000006a8f2000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:402)
> - waiting to lock <0x00002aaab46f8938> (a
> org.apache.activemq.broker.region.Topic)
> at org.apache.activemq.broker.region.Topic.send(Topic.java:384)
> at
> org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:93)
> at
> org.apache.activemq.broker.region.virtual.VirtualTopicInterceptor.send(VirtualTopicInterceptor.java:46)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:443)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:373)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:368)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:173)
> at
> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:214)
> at
> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:214)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:221)
> at
> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:467)
> at
> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:82)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:135)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.createTempQueue(ProtocolConverter.java:611)
> at
> org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertDestination(LegacyFrameTranslator.java:130)
> at
> org.apache.activemq.transport.stomp.FrameTranslator$Helper.copyStandardHeadersFromFrameToMessage(FrameTranslator.java:93)
> at
> org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertFrame(LegacyFrameTranslator.java:54)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:591)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompSend(ProtocolConverter.java:231)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:173)
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:71)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
> at java.lang.Thread.run(Thread.java:619)
> It is waiting to lock 0x00002aaab46f8938. This is the thread that holds the
> lock:
> "ActiveMQ Transport: tcp:///10.231.233.117:43020" daemon prio=10
> tid=0x00000000608d8800 nid=0xc6e in Object.wait() [0x000000007d41c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:106)
> - locked <0x00002aaab57d34e8> (a java.lang.Object)
> at
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> - locked <0x00002aaab46f98e0> (a
> java.util.concurrent.CopyOnWriteArrayList)
> at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:585)
> at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:443)
> - locked <0x00002aaab46f8938> (a
> org.apache.activemq.broker.region.Topic)
> at org.apache.activemq.broker.region.Topic.send(Topic.java:384)
> at
> org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:93)
> at
> org.apache.activemq.broker.region.virtual.VirtualTopicInterceptor.send(VirtualTopicInterceptor.java:46)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:443)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:373)
> at
> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:368)
> at
> org.apache.activemq.advisory.AdvisoryBroker.removeDestination(AdvisoryBroker.java:185)
> at
> org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:146)
> at
> org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:146)
> at
> org.apache.activemq.broker.MutableBrokerFilter.removeDestination(MutableBrokerFilter.java:153)
> at
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:713)
> - locked <0x00002aaac32b10c0> (a
> org.apache.activemq.broker.jmx.ManagedTransportConnection)
> at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:72)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:82)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:135)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompDisconnect(ProtocolConverter.java:545)
> at
> org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:189)
> at
> org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:71)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
> at java.lang.Thread.run(Thread.java:619)
> So it is the client 10.231.233.117 that has a stomp-connection to the broker,
> causes a TopicSubscription.add and causes the broker freeze.
> This is how we configure our openwire transport-connector in the activemq.xml:
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
> <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> </transportConnectors>
> We took jstack-dumps with a distance of 9 minutes. Both are attached. In both
> dumps it is the same TopicSubscription.add that causes the hang.
> How can we solve this issue? Looks like the connection should run into an
> timeout but doesn't. Is this an activemq bug? Can we somehow set the broker
> openwire timeout parameters?
> Thanks for any hints.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
