it's possible to cause a deadlock in ActiveMQ 5.2 using the client
------------------------------------------------------------------
Key: AMQ-2332
URL: https://issues.apache.org/activemq/browse/AMQ-2332
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.2.0
Reporter: Mats Henrikson
Hi,
We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing
it by a multi-threading problem in the client, we are currently looking into
this. However, I feel that this is still a bug in the broker since it shouldn't
be possible to deadlock the broker by incorrect use of the client.
The situation is that we have two clients connected to the broker, one over TCP
using the Java library, and one connecting over SSL (our own implementation,
this may also be involved in creating the problem) using the NMS library in
.NET.
The Java client connects fine and subscribes to some topics and queues and
starts sending messages to the topics. The NMS client connects and attempts to
connect to some of the same topics, as well as creating a temporary queue. As
it is doing this initial setup stuff it hangs and goes no further. From this
point I don't think any more messages get delivered on the topic, and Java
clients trying to send messages to it hang on the producers send() call
indefinitely.
The deadlock appears to occur because a thread is stuck actually writing data
back to the NMS client (the bottom stack trace below), another thread
(originating from the Java client) is trying to acquire the write lock so that
it too can write to the client (middle stack trace) while still owning a lock
on the list of consumers of a topic, meanwhile a third thread (this one
origination from the NMS client again) is trying to get that lock on the list
of consumers in order to add to it (top stack trace). For some reason the first
thread trying to write to the NMS client never finishes, even when the NMS
client is killed, and so the lock is never released and everything involving
that topic grinds to a halt.
I have looked but haven't been able to find anybody else reporting this issue.
I enabled JMX as well as debug mode and got the following stack traces from
jconsole, and tracked down the locks involved in the deadlock using a debugger:
Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
State: BLOCKED on java.util.concurrent.copyonwritearrayl...@509c6c30 owned by:
ActiveMQ Transport: tcp:///192.168.32.73:45666
Stack trace:
org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
copyonwritearrayl...@509c6c30 lock acquired at
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
below:
Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
State: BLOCKED on java.lang.obj...@58c1c2f1 owned by: TempQueue:
c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
Stack trace:
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
org.apache.activemq.broker.region.Topic.send(Topic.java:365)
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
obj...@58c1c2f1 lock acquired at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
below:
Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
State: RUNNABLE
Total blocked: 55 Total waited: 11
Stack trace:
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Let me know if I can do anything else to help.
--
Mats Henrikson
Software Engineer
SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
