[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-12-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296172#comment-16296172
 ] 

ASF GitHub Bot commented on AMQ-6601:
-

Github user asfgit closed the pull request at:

https://github.com/apache/activemq/pull/228


> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
>Assignee: Hadrian Zbarcea
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> - waiting to lock <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-12-18 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296171#comment-16296171
 ] 

ASF subversion and git services commented on AMQ-6601:
--

Commit b40e5f93bdfdecad84f880edce23c4ae8af9a76d in activemq's branch 
refs/heads/master from [~catull]
[ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=b40e5f9 ]

[AMQ-6601] AMQ with master and slave broker, shutting down slave causes 
dead-lock.


> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
>Assignee: Hadrian Zbarcea
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
>

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-12-18 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296167#comment-16296167
 ] 

ASF subversion and git services commented on AMQ-6601:
--

Commit 30edca5b22d95e919f403c03b1c307914572ba65 in activemq's branch 
refs/heads/activemq-5.14.x from [~catull]
[ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=30edca5 ]

[AMQ-6601] AMQ with master and slave broker, shutting down slave causes 
dead-lock.


> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
>Assignee: Hadrian Zbarcea
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
>

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-12-18 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296166#comment-16296166
 ] 

ASF subversion and git services commented on AMQ-6601:
--

Commit b04b9716855d95eae0276cc56c464765d3fbd95e in activemq's branch 
refs/heads/activemq-5.15.x from [~catull]
[ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=b04b971 ]

[AMQ-6601] AMQ with master and slave broker, shutting down slave causes 
dead-lock.


> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
>Assignee: Hadrian Zbarcea
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
>

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-05-30 Thread Martin Lichtin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029443#comment-16029443
 ] 

Martin Lichtin commented on AMQ-6601:
-

Upgrading to 5.14.5 I noticed the issue is still there. Can the change be 
included in the next release?


> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> - waiting to lock <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-03-04 Thread Carlo Dapor (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895849#comment-15895849
 ] 

Carlo Dapor commented on AMQ-6601:
--

Created pull request https://github.com/apache/activemq/pull/228.

> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at 
> org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x7f79d0c48800 nid=0x7080 waiting on 
> condition [0x7f79985b5000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x7f79d0c48000 
> nid=0x707f waiting for monitor entry [0x7f79986b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> - waiting to lock <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
>

[jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock

2017-03-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895846#comment-15895846
 ] 

ASF GitHub Bot commented on AMQ-6601:
-

GitHub user catull opened a pull request:

https://github.com/apache/activemq/pull/228

[AMQ-6601] AMQ with master and slave broker, shutting down slave causes a 
dead-lock

This PR addresses the issue https://issues.apache.org/jira/browse/AMQ-6601.

The proposed solution is to start the broker asynchronously, but only wait 
for its starting up if it is the master (i.e. non-slave).

Closes / fixes issue #6601.

Credit goes to @lichtin for coming up with the solution.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/catull/activemq 
fix-issue-6601-shutdown-slave-broker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq/pull/228.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #228


commit 5de4f788d17f406c713d42a1bd77c15b973ac0de
Author: Carlo Dapor 
Date:   2017-03-04T20:01:34Z

[AMQ-6601] AMQ with master and slave broker, shutting down slave causes 
dead-lock.




> AMQ with master and slave broker, shutting down slave causes dead-lock
> --
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.0
>Reporter: Carlo Dapor
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) 
> and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - 
> hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when 
> the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: 
> factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 
> tid=0x7f793c160800 nid=0x7084 waiting on condition [0x7f799819f000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at 
> org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at 
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at 
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at 
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at 
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at 
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at 
> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x00072bd74db0> (a 
> org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at 
> org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at 
> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x7f7940002800 nid=0x7081 
> runnable [0x7f79984b4000]
>java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at