[
https://issues.apache.org/jira/browse/AMQ-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589553#comment-13589553
]
Frederik Colardyn edited comment on AMQ-4351 at 2/28/13 2:19 PM:
-----------------------------------------------------------------
Hi Gary,
It's easy to reproduce, and we've got a unit test for it (see attach), but it's
not very well suited to include in an automated build system as it produces
high load over ± 4 minutes in order to trigger the deadlock. It's reliable
though, always deadlocking without the patch.
Scenario:
1. 100 durable consumers, of which 97 go inactive after receiving the first
message
2. producer sends messages as fast as possible
3. after 3 minutes, broker tries to clean up the 97 inactive producers =>
always deadlocks. No more messages are being sent to the active produces. kill
-3 of the jms broker always shows the same deadlock.
We took 100 durable consumers in our test in order to provoke the deadlock
quickly, but in our production system it occurred with only 4 consumers (of
which 1 was cleaned up after 1 hour inactivity).
was (Author: frederikcolardyn):
Hi Gary,
It's easy to reproduce, and we've got a unit test for it (see attach), but it's
not very well suited to include in an automated build system as it produces
high load over ± 4 minutes in order to trigger the deadlock. It's reliable
though, always deadlocking without the patch.
Scenario:
1. 100 durable consumers, of which 97 go inactive after receiving the first
message
2. producer sends messages as fast as possible
3. after 3 minutes, broker tries to clean up the 97 inactive producers =>
always deadlocks. No more messages are being sent at all.
We took 100 durable consumers in our test in order to provoke the deadlock
quickly, but in our production system it occurred with only 4 consumers (of
which 1 was cleaned up after 1 hour inactivity).
> Deadlock when unsubscribing durable subscriber
> ----------------------------------------------
>
> Key: AMQ-4351
> URL: https://issues.apache.org/jira/browse/AMQ-4351
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.7.0, 5.8.0
> Environment: Setup:
> Active MQ 5.7.0 with master/slave using JDBC store
> Approx 3 - 5 concurrent durable subscribers
> Approx 5 messages / sec
> Reporter: Tim Lammens
> Labels: patch
> Attachments: ActiveMQUnsubscribeTest.java, dispatcher.patch
>
>
> We started using durable subscribers a week ago, and after the 4th durable
> subscriber unsubscribed (due to 1 hour inactivity), the system deadlocked. If
> the "Durable Subscriber Cleanup Timer" goes of at the wrong time, your entire
> server locks up.
> Setup:
> Active MQ 5.7.0 with master/slave using JDBC store
> Approx 3 - 5 concurrent durable subscribers
> Approx 5 messages / sec
> Active MQ checks every 1 minute for subscribers which have been offline for 1
> hour.
> Locked threads:
> "ActiveMQ Transport: tcp:///79.125.71.104:48082@8090":
> at
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:446)
> - waiting to lock <0x00000000c6228480> (a
> org.apache.activemq.broker.region.Topic)
> at org.apache.activemq.broker.region.Topic.send(Topic.java:427)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
>
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
> "ActiveMQ Transport: tcp:///79.125.71.104:47590@8090":
> at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:142)
>
> - waiting to lock <0x00000000c66ba050> (a java.lang.Object)
> at
> org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:243)
>
> at
> org.apache.activemq.broker.region.policy.StrictOrderDispatchPolicy.dispatch(StrictOrderDispatchPolicy.java:58)
>
> "ActiveMQ Durable Subscriber Cleanup Timer":
> at org.apache.activemq.broker.region.Topic.deactivate(Topic.java:288)
> - waiting to lock <0x00000000c6250670> (a
> java.util.concurrent.CopyOnWriteArrayList)
> at
> org.apache.activemq.broker.region.DurableTopicSubscription.deactivate(DurableTopicSubscription.java:184)
>
> - locked <0x00000000c66ba060> (a java.lang.Object)
> - locked <0x00000000c66ba050> (a java.lang.Object)
> at
> org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:195)
> at
> org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:199)
>
> at
> org.apache.activemq.broker.region.TopicRegion.doCleanup(TopicRegion.java:99)
> I have attached a patch which fixes the problem.
> Since there is only one dispatch policy per Topic, synchronisation can happen
> on the DispatchPolicy instead of on the consumers object which causes the
> deadlock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
