[ https://issues.apache.org/jira/browse/QPID-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Philip Harvey updated QPID-4276: -------------------------------- Description: We have observed a deadlock on the Java client when the following are done concurrently: * An application thread closes a consumer * The corresponding JMS connection is closed inside onMessage() See attached stack trace for details. We think this locking bug was introduced when fixing QPID-3911. See thread dump excerpt below: {noformat} Found one Java-level deadlock: ============================= "Thread-11-Consumer_0": waiting to lock monitor 0x000000004342b4c0 (object 0x00000000ed1e6a40, a java.lang.Object), which is held by "Dispatcher-1-Conn-1" "Dispatcher-1-Conn-1": waiting to lock monitor 0x000000004342b568 (object 0x00000000ed060c48, a java.lang.Object), which is held by "Thread-11-Consumer_0" Java stack information for the threads listed above: =================================================== "Thread-11-Consumer_0": at org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:598) - waiting to lock <0x00000000ed1e6a40> (a java.lang.Object) - locked <0x00000000ed060c48> (a java.lang.Object) at org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:558) at org.apache.qpid.disttest.jms.ClientJmsDelegate.closeTestConsumer(ClientJmsDelegate.java:612) at org.apache.qpid.disttest.client.ConsumerParticipant.releaseResources(ConsumerParticipant.java:244) at org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.runParticipantAndSendResults(ParticipantExecutor.java:112) at org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.run(ParticipantExecutor.java:84) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "Dispatcher-1-Conn-1": at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:857) - waiting to lock <0x00000000ed060c48> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) - locked <0x00000000ed0bedd8> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) - locked <0x00000000ed1e6a40> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:832) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:823) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:818) at org.apache.qpid.disttest.jms.ClientJmsDelegate.tearDownTest(ClientJmsDelegate.java:574) at org.apache.qpid.disttest.client.Client.tearDownTest(Client.java:179) at org.apache.qpid.disttest.client.ClientCommandVisitor.visit(ClientCommandVisitor.java:90) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.qpid.disttest.Visitor.invokeVisitMethod(Visitor.java:61) at org.apache.qpid.disttest.Visitor.visit(Visitor.java:39) at org.apache.qpid.disttest.message.Command.accept(Command.java:48) at org.apache.qpid.disttest.client.Client.processInstruction(Client.java:113) at org.apache.qpid.disttest.jms.ClientJmsDelegate$1.onMessage(ClientJmsDelegate.java:122) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:751) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:725) at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3479) at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3418) - locked <0x00000000ed1a30f0> (a java.lang.Object) - locked <0x00000000ed186400> (a java.lang.Object) at org.apache.qpid.client.AMQSession$Dispatcher.access$1000(AMQSession.java:3205) at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3198) at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3341) at java.lang.Thread.run(Thread.java:662) Found 1 deadlock. {noformat} was: We have observed a deadlock on the Java client when a consumer and its connection are simultaneously closed. See attached stack trace for details. We think this locking bug was introduced when fixing QPID-3911. See thread dump excerpt below: {noformat} Found one Java-level deadlock: ============================= "Thread-11-Consumer_0": waiting to lock monitor 0x000000004342b4c0 (object 0x00000000ed1e6a40, a java.lang.Object), which is held by "Dispatcher-1-Conn-1" "Dispatcher-1-Conn-1": waiting to lock monitor 0x000000004342b568 (object 0x00000000ed060c48, a java.lang.Object), which is held by "Thread-11-Consumer_0" Java stack information for the threads listed above: =================================================== "Thread-11-Consumer_0": at org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:598) - waiting to lock <0x00000000ed1e6a40> (a java.lang.Object) - locked <0x00000000ed060c48> (a java.lang.Object) at org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:558) at org.apache.qpid.disttest.jms.ClientJmsDelegate.closeTestConsumer(ClientJmsDelegate.java:612) at org.apache.qpid.disttest.client.ConsumerParticipant.releaseResources(ConsumerParticipant.java:244) at org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.runParticipantAndSendResults(ParticipantExecutor.java:112) at org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.run(ParticipantExecutor.java:84) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "Dispatcher-1-Conn-1": at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:857) - waiting to lock <0x00000000ed060c48> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) - locked <0x00000000ed0bedd8> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) - locked <0x00000000ed1e6a40> (a java.lang.Object) - locked <0x00000000ed060c58> (a java.lang.Object) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:832) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:823) at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:818) at org.apache.qpid.disttest.jms.ClientJmsDelegate.tearDownTest(ClientJmsDelegate.java:574) at org.apache.qpid.disttest.client.Client.tearDownTest(Client.java:179) at org.apache.qpid.disttest.client.ClientCommandVisitor.visit(ClientCommandVisitor.java:90) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.qpid.disttest.Visitor.invokeVisitMethod(Visitor.java:61) at org.apache.qpid.disttest.Visitor.visit(Visitor.java:39) at org.apache.qpid.disttest.message.Command.accept(Command.java:48) at org.apache.qpid.disttest.client.Client.processInstruction(Client.java:113) at org.apache.qpid.disttest.jms.ClientJmsDelegate$1.onMessage(ClientJmsDelegate.java:122) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:751) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:725) at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3479) at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3418) - locked <0x00000000ed1a30f0> (a java.lang.Object) - locked <0x00000000ed186400> (a java.lang.Object) at org.apache.qpid.client.AMQSession$Dispatcher.access$1000(AMQSession.java:3205) at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3198) at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3341) at java.lang.Thread.run(Thread.java:662) Found 1 deadlock. {noformat} > Java client - deadlock on concurrent close of consumer and connection > --------------------------------------------------------------------- > > Key: QPID-4276 > URL: https://issues.apache.org/jira/browse/QPID-4276 > Project: Qpid > Issue Type: Bug > Components: Java Client > Affects Versions: 0.16, 0.18 > Reporter: Philip Harvey > Attachments: deadlock.stacktrace.txt > > > We have observed a deadlock on the Java client when the following are done > concurrently: > * An application thread closes a consumer > * The corresponding JMS connection is closed inside onMessage() > See attached stack trace for details. > We think this locking bug was introduced when fixing QPID-3911. > See thread dump excerpt below: > {noformat} > Found one Java-level deadlock: > ============================= > "Thread-11-Consumer_0": > waiting to lock monitor 0x000000004342b4c0 (object 0x00000000ed1e6a40, a > java.lang.Object), > which is held by "Dispatcher-1-Conn-1" > "Dispatcher-1-Conn-1": > waiting to lock monitor 0x000000004342b568 (object 0x00000000ed060c48, a > java.lang.Object), > which is held by "Thread-11-Consumer_0" > Java stack information for the threads listed above: > =================================================== > "Thread-11-Consumer_0": > at > org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:598) > - waiting to lock <0x00000000ed1e6a40> (a java.lang.Object) > - locked <0x00000000ed060c48> (a java.lang.Object) > at > org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:558) > at > org.apache.qpid.disttest.jms.ClientJmsDelegate.closeTestConsumer(ClientJmsDelegate.java:612) > at > org.apache.qpid.disttest.client.ConsumerParticipant.releaseResources(ConsumerParticipant.java:244) > at > org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.runParticipantAndSendResults(ParticipantExecutor.java:112) > at > org.apache.qpid.disttest.client.ParticipantExecutor$ParticipantRunnable.run(ParticipantExecutor.java:84) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "Dispatcher-1-Conn-1": > at > org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:857) > - waiting to lock <0x00000000ed060c48> (a java.lang.Object) > - locked <0x00000000ed060c58> (a java.lang.Object) > at > org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) > - locked <0x00000000ed0bedd8> (a java.lang.Object) > - locked <0x00000000ed060c58> (a java.lang.Object) > at > org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:848) > - locked <0x00000000ed1e6a40> (a java.lang.Object) > - locked <0x00000000ed060c58> (a java.lang.Object) > at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:832) > at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:823) > at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:818) > at > org.apache.qpid.disttest.jms.ClientJmsDelegate.tearDownTest(ClientJmsDelegate.java:574) > at > org.apache.qpid.disttest.client.Client.tearDownTest(Client.java:179) > at > org.apache.qpid.disttest.client.ClientCommandVisitor.visit(ClientCommandVisitor.java:90) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.qpid.disttest.Visitor.invokeVisitMethod(Visitor.java:61) > at org.apache.qpid.disttest.Visitor.visit(Visitor.java:39) > at org.apache.qpid.disttest.message.Command.accept(Command.java:48) > at > org.apache.qpid.disttest.client.Client.processInstruction(Client.java:113) > at > org.apache.qpid.disttest.jms.ClientJmsDelegate$1.onMessage(ClientJmsDelegate.java:122) > at > org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:751) > at > org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:725) > at > org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3479) > at > org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3418) > - locked <0x00000000ed1a30f0> (a java.lang.Object) > - locked <0x00000000ed186400> (a java.lang.Object) > at > org.apache.qpid.client.AMQSession$Dispatcher.access$1000(AMQSession.java:3205) > at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3198) > at > org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54) > at > org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3341) > at java.lang.Thread.run(Thread.java:662) > Found 1 deadlock. > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org