[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-12-01 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034322#comment-15034322
 ] 

Pablo Lozano commented on AMQ-6052:
---

Is there anything missing to check this issue?

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.10.1, 5.12.1, 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> 

[jira] [Created] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-18 Thread Pablo Lozano (JIRA)
Pablo Lozano created AMQ-6052:
-

 Summary: Network of brokers on duplex mode reports 
InstanceAlreadyExistsException on already existing destinations
 Key: AMQ-6052
 URL: https://issues.apache.org/jira/browse/AMQ-6052
 Project: ActiveMQ
  Issue Type: Bug
  Components: networkbridge
Affects Versions: 5.11.3
 Environment: Linux
Reporter: Pablo Lozano


When using a network of brokers apparently on duplex with destinations which 
were already created on each MBeanBridgeDestination.onOutboundMessage() and 
MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
MBean of a destination which has already been created.

Here is a discussion that started but a ticket was not created.
http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html

Although this does not seem to impact the functionality of the application it 
creates a massive amount of logs as this message repeats for every received 
message. 

This are the important bits of my activeMQ configuration:
{code:xml}
   


























{code}

And this is the output log generated:
{noformat}
2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination 
- Failed to register queue://mailsystem.templateprocessor
javax.management.InstanceAlreadyExistsException: 
org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at 
org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
at 
org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
at 
org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
at 
org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
at 
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
at 
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
at 
org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
at 
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at 
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at 
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at 
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
at 
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at 
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at 
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)
{noformat}

{noformat}
2015-11-18 15:43:09,634 [Broker] Task-16] WARN  
MBeanBridgeDestination - Failed to register 
queue://mailsystem.templateprocessor
javax.management.InstanceAlreadyExistsException: 
org.apache.activemq:brokerName=mailsystemBroker,connector=networkConnectors,networkConnectorName=mailSystemConnector,networkBridge=tcp_//10.211.2.70_36731,type=Broker,direction=outbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at 

[jira] [Updated] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

 [ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Lozano updated AMQ-6052:
--
Attachment: AMQ_6052.patch

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
> at 
> 

[jira] [Comment Edited] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018499#comment-15018499
 ] 

Pablo Lozano edited comment on AMQ-6052 at 11/20/15 6:45 PM:
-

I have included a patch for this issue. From the way that it is done it removes 
the need to synchronize operations which is a good thing.


was (Author: altaflux):
I have included a patch for this issue.

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> 

[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018499#comment-15018499
 ] 

Pablo Lozano commented on AMQ-6052:
---

I have included a patch for this issue.

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> 

[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018701#comment-15018701
 ] 

Pablo Lozano commented on AMQ-6052:
---

No problem, 
Let me work out a test and attach it.

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> 

[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-19 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015059#comment-15015059
 ] 

Pablo Lozano commented on AMQ-6052:
---

I have found the issue that activates this problem.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
private Map destinationObjectNameMap = new 
ConcurrentHashMap();
private Map 
outboundDestinationViewMap = new ConcurrentHashMap();
private Map 
inboundDestinationViewMap = new ConcurrentHashMap();
{code}

Both onOutboundMessage() and onInboundMessage() use the  
destinationObjectNameMap to register the ObjectName of the 
NetworkDestinationView. But each other overwrite their value when using a 
duplex connection.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
public void onInboundMessage(Message message) {

 destinationObjectNameMap.put(destination, objectName);
}
{code}

When running with the option to purge unactiveDestinationViews which by default 
is active it will un-register from the MBeanServer the objectName that is found 
on  destinationObjectNameMap.remove(entry.getKey()); //ActiveMqDestination. 
This happens on the method 
purgeInactiveDestinationView(Map 
map) which receives both outboundDestinationViewMap and 
inboundDestinationViewMap.

Thus when running for outboundDestinationViewMap if the last registration of a 
destination was actually of a inBound it will incorrectly delete  the object 
name of and  inboundDestination insead of the one from outboundDestination.

I am working on a fix and submitting a patch for this issue.



> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> 

[jira] [Comment Edited] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-19 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015059#comment-15015059
 ] 

Pablo Lozano edited comment on AMQ-6052 at 11/20/15 2:51 AM:
-

I have found the issue that activates this problem.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
private Map destinationObjectNameMap = new 
ConcurrentHashMap();
private Map 
outboundDestinationViewMap = new ConcurrentHashMap();
private Map 
inboundDestinationViewMap = new ConcurrentHashMap();
{code}

Both onOutboundMessage() and onInboundMessage() use the  
destinationObjectNameMap to register the ObjectName of the 
NetworkDestinationView. But each other overwrite their value when using a 
duplex connection.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
public void onInboundMessage(Message message) {

 destinationObjectNameMap.put(destination, objectName);
}
{code}

When running with the option to purge unactiveDestinationViews which by default 
is active it will un-register from the MBeanServer the objectName that is found 
on  destinationObjectNameMap.remove(entry.getKey()); //ActiveMqDestination. 
This happens on the method 
purgeInactiveDestinationView(Map 
map) which receives both outboundDestinationViewMap and 
inboundDestinationViewMap.

Thus when running for outboundDestinationViewMap if the last registration of a 
destination was actually of a inBound it will incorrectly delete  the object 
name of and  inboundDestination insead of the one from outboundDestination.

I am working on a fix and submitting a patch for this issue.

This seems to be related to AMQ-5265 which looks like it tried to fix the same 
issue although apparently it was still present.


was (Author: altaflux):
I have found the issue that activates this problem.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
private Map destinationObjectNameMap = new 
ConcurrentHashMap();
private Map 
outboundDestinationViewMap = new ConcurrentHashMap();
private Map 
inboundDestinationViewMap = new ConcurrentHashMap();
{code}

Both onOutboundMessage() and onInboundMessage() use the  
destinationObjectNameMap to register the ObjectName of the 
NetworkDestinationView. But each other overwrite their value when using a 
duplex connection.

{code:title=MBeanBridgeDestination.java|borderStyle=solid}
public void onInboundMessage(Message message) {

 destinationObjectNameMap.put(destination, objectName);
}
{code}

When running with the option to purge unactiveDestinationViews which by default 
is active it will un-register from the MBeanServer the objectName that is found 
on  destinationObjectNameMap.remove(entry.getKey()); //ActiveMqDestination. 
This happens on the method 
purgeInactiveDestinationView(Map 
map) which receives both outboundDestinationViewMap and 
inboundDestinationViewMap.

Thus when running for outboundDestinationViewMap if the last registration of a 
destination was actually of a inBound it will incorrectly delete  the object 
name of and  inboundDestination insead of the one from outboundDestination.

I am working on a fix and submitting a patch for this issue.



> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this 

[jira] [Updated] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

 [ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Lozano updated AMQ-6052:
--
Patch Info: Patch Available

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
>

[jira] [Updated] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

 [ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Lozano updated AMQ-6052:
--
Attachment: AMQ_6052-2.patch

Patch with test case included

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> 

[jira] [Updated] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2015-11-20 Thread Pablo Lozano (JIRA)

 [ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Lozano updated AMQ-6052:
--
Affects Version/s: 5.10.1
   5.12.1

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.10.1, 5.12.1, 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>  Labels: jmx, networkBridge, networkConnector
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
> at 
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> at 
> 

[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2016-01-11 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092338#comment-15092338
 ] 

Pablo Lozano commented on AMQ-6052:
---

Great thanks Gary,
I expect it to work correctly, we have run this patch in production for a while 
and haven't seen the issue occur again so fingers crossed.

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.10.1, 5.12.1, 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>Assignee: Gary Tully
>  Labels: jmx, networkBridge, networkConnector
> Fix For: 5.14.0
>
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:649)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport$3.onCommand(DemandForwardingBridgeSupport.java:224)
> at 
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at 
> 

[jira] [Commented] (AMQ-6052) Network of brokers on duplex mode reports InstanceAlreadyExistsException on already existing destinations

2016-01-19 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107363#comment-15107363
 ] 

Pablo Lozano commented on AMQ-6052:
---

This issue can also be worked around be setting on the networkConnector 
gcDestinationViews="false".

Although disabling the log level on that class avoids the huge log output it 
still creates a performance impact during the time where the issue occurs until 
the next removal of destinations as for every message that is received being on 
that faulty state will enter the synchronization block and throw and exception 
which will cause performance to degrade.

> Network of brokers on duplex mode reports InstanceAlreadyExistsException on 
> already existing destinations
> -
>
> Key: AMQ-6052
> URL: https://issues.apache.org/jira/browse/AMQ-6052
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: networkbridge
>Affects Versions: 5.10.1, 5.12.1, 5.11.3
> Environment: Linux
>Reporter: Pablo Lozano
>Assignee: Gary Tully
>  Labels: jmx, networkBridge, networkConnector
> Fix For: 5.13.1, 5.14.0, 5.12.3
>
> Attachments: AMQ_6052-2.patch, AMQ_6052.patch
>
>
> When using a network of brokers apparently on duplex with destinations which 
> were already created on each MBeanBridgeDestination.onOutboundMessage() and 
> MBeanBridgeDestination.onInboundMessage() the bridge tries to register the 
> MBean of a destination which has already been created.
> Here is a discussion that started but a ticket was not created.
> http://activemq.2283324.n4.nabble.com/Broker-log-full-of-Failed-to-register-queue-messages-td4685241.html
> Although this does not seem to impact the functionality of the application it 
> creates a massive amount of logs as this message repeats for every received 
> message. 
> This are the important bits of my activeMQ configuration:
> {code:xml}
>
>  uri="multicast://default" conduitSubscriptions="true" duplex="true" >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> connectorPort="1091"
>jmxDomainName="org.apache.activemq"/>
> 
> 
> 
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  discoveryUri="multicast://default" 
> updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
>  updateClusterClients="true"
> rebalanceClusterClients="true" 
> updateClusterClientsOnRemove="true"/>
> 
> {code}
> And this is the output log generated:
> {noformat}
> 2015-11-18 15:43:10,497 [.69:41090@36731] WARN  MBeanBridgeDestination
>  - Failed to register queue://mailsystem.templateprocessor
> javax.management.InstanceAlreadyExistsException: 
> org.apache.activemq:brokerName=mailsystemBroker,connector=duplexNetworkConnectors,networkConnectorName=#0,networkBridge=tcp_//10.211.2.69_41090,type=Broker,direction=inbound,destinationType=Queue,destinationName=mailsystem.templateprocessor
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at 
> org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)
> at 
> org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)
> at 
> org.apache.activemq.network.MBeanBridgeDestination.onInboundMessage(MBeanBridgeDestination.java:97)
> at 
> org.apache.activemq.network.MBeanNetworkListener.onInboundMessage(MBeanNetworkListener.java:115)
> at 
> org.apache.activemq.network.DemandForwardingBridgeSupport.serviceInboundMessage(DemandForwardingBridgeSupport.java:1680)
> at 
> 

[jira] [Commented] (AMQ-5618) Infinite loop in log replay with Replicated LevelDB

2017-02-06 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854412#comment-15854412
 ] 

Pablo Lozano commented on AMQ-5618:
---

Hey [~cshannon] could you do me a favor and point me to a place where I can 
find the discussion in which LevelDB was decided to be deprecated? Mailing List 
archive or something similar would be great.

Thanks

> Infinite loop in log replay with Replicated LevelDB
> ---
>
> Key: AMQ-5618
> URL: https://issues.apache.org/jira/browse/AMQ-5618
> Project: ActiveMQ
>  Issue Type: Bug
>Affects Versions: 5.11.0, 5.11.1
> Environment: Linux, Google Compute Engine
>Reporter: Artem Karpenko
>Priority: Critical
>
> This is very similar to AMQ-5300 except that I use replicatedLevelDB 
> persistence adapter and in order to reproduce I don't have to delete any 
> index files.
> Setup: 1 ZK instance, 3 AMQ nodes.
> One of the AMQ configs:
> {code}
>  replicas="3"
> bind="tcp://0.0.0.0:61619"
> zkAddress="instance-6:2181"
> zkPath="/activemq/leveldb-stores"
> hostname="instance-7" />
> {code}
> Difference between nodes is only in hostname attribute.
> The way to reproduce is almost the same as in AMQ-5300: 
> # Produce lots of messages to generate several log files in leveldb data 
> directory.
> # Consume _some_ messages until you see "Deleting log" in activemq.log.
> # Restart master. Wait for system to rebalance itself. Everything's fine at 
> this point.
> # Restart the second master.
> # Observe the massive (infinite?) logging on slave and relatively calm but 
> still possibly infinite logging on master.
> This is what the first master logs after it's restarted:
> {code}
> 2015-02-25 21:37:08,338 | DEBUG | Download session connected... | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:08,582 | INFO  | Slave skipping download of: 
> log/190be289.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,099 | INFO  | Slave skipping download of: 
> log/0642f848.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,411 | INFO  | Slave skipping download of: 
> log/0c85f06d.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,838 | INFO  | Slave skipping download of: 
> log/12c8e921.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,842 | INFO  | Slave requested: 
> 1c9373b4.index/CURRENT | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,846 | INFO  | Slave requested: 
> 1c9373b4.index/MANIFEST-02 | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,850 | INFO  | Slave requested: 
> 1c9373b4.index/03.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,857 | INFO  | Attaching... Downloaded 0.02/95.65 kb and 
> 1/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,859 | INFO  | Attaching... Downloaded 0.06/95.65 kb and 
> 2/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,861 | INFO  | Attaching... Downloaded 95.65/95.65 kb and 
> 3/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,862 | INFO  | Attached | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,878 | DEBUG | Taking a snapshot of the current index: 
> /usr/local/apache-activemq-5.11.1/data/replicatedLevelDB/1c9373b4.index
>  | org.apache.activemq.leveldb.LevelDBClient | Thread-2
> 2015-02-25 21:37:10,352 | DEBUG | Recovering from last index snapshot at: 
> /usr/local/apache-activemq-5.11.1/data/replicatedLevelDB/dirty.index | 
> org.apache.activemq.leveldb.LevelDBClient | Thread-2
> {code}
> Right after that everything seems fine. But as soon as I stop the new master, 
> the another new master (that would be the third one) logs
> {code}
> 2015-02-25 21:38:43,876 | INFO  | Promoted to master | 
> org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> 2015-02-25 21:38:43,894 | INFO  | Using the pure java LevelDB implementation. 
> | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ 
> BrokerService[localhost] Task-5
> 

[jira] [Commented] (AMQ-5618) Infinite loop in log replay with Replicated LevelDB

2016-09-22 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514423#comment-15514423
 ] 

Pablo Lozano commented on AMQ-5618:
---

I have done some digging around.. and it seems that it will sooner or later 
will stop the loop.. eventually at least.
I dont have the setup of Replicated LevelDB at hand right now but from I can 
see is that is basically lost track about the last position on the records of 
LevelDB.

>From the logs in my case it would try to iterate from 0 to the last position 
>it knows exists. So in my case it would iterate until 32610993430.
The fix from AMQ-5300 seems not to be executed as this replay of the logs of 
LevelDB is being triggered by a call which bypasses its fix. The fix basically 
is not to start replaying from 0 but from the last record registered.

{noformat}
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] LevelDBClient  - Could not load message 
seq: 162542 from DataLocator(11e952fc, 1754)
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] RecordLog  - No reader available for 
position: 11e999ce, log_infos: 
{32610993430=LogInfo(/m2/tomcat7.0/work/navigator-mail-mq/6009/work/repDB/000797c44516.log,32610993430,0)}
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] LevelDBClient  - Could not load message 
seq: 162552 from DataLocator(11e999ce, 1754)
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] RecordLog  - No reader available for 
position: 11e9a7f8, log_infos: 
{32610993430=LogInfo(/m2/tomcat7.0/work/navigator-mail-mq/6009/work/repDB/000797c44516.log,32610993430,0)}
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] LevelDBClient  - Could not load message 
seq: 162554 from DataLocator(11e9a7f8, 1754)
2015-01-29 19:42:55,740 -0600 WARN  56426 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] RecordLog  - No reader available for 
position: 11e9b622, log_infos: 
{32610993430=LogInfo(/m2/tomcat7.0/work/navigator-mail-mq/6009/work/repDB/000797c44516.log,32610993430,0)}
2015-01-29 19:42:55,741 -0600 WARN  56427 [ActiveMQ 
BrokerService[mailsystemBroker] Task-2] LevelDBClient  - Could not load message 
seq: 162556 from DataLocator(11e9b622, 1754)
{noformat}

I lost most of my log files from this issue but if someone can attach then if 
possible at Trace level and if possible with a LevelDB included I may be able 
to take a look deeper. My Scala is not the best but I think I can help track 
the root of the issue

> Infinite loop in log replay with Replicated LevelDB
> ---
>
> Key: AMQ-5618
> URL: https://issues.apache.org/jira/browse/AMQ-5618
> Project: ActiveMQ
>  Issue Type: Bug
>Affects Versions: 5.11.0, 5.11.1
> Environment: Linux, Google Compute Engine
>Reporter: Artem Karpenko
>Priority: Critical
>
> This is very similar to AMQ-5300 except that I use replicatedLevelDB 
> persistence adapter and in order to reproduce I don't have to delete any 
> index files.
> Setup: 1 ZK instance, 3 AMQ nodes.
> One of the AMQ configs:
> {code}
>  replicas="3"
> bind="tcp://0.0.0.0:61619"
> zkAddress="instance-6:2181"
> zkPath="/activemq/leveldb-stores"
> hostname="instance-7" />
> {code}
> Difference between nodes is only in hostname attribute.
> The way to reproduce is almost the same as in AMQ-5300: 
> # Produce lots of messages to generate several log files in leveldb data 
> directory.
> # Consume _some_ messages until you see "Deleting log" in activemq.log.
> # Restart master. Wait for system to rebalance itself. Everything's fine at 
> this point.
> # Restart the second master.
> # Observe the massive (infinite?) logging on slave and relatively calm but 
> still possibly infinite logging on master.
> This is what the first master logs after it's restarted:
> {code}
> 2015-02-25 21:37:08,338 | DEBUG | Download session connected... | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:08,582 | INFO  | Slave skipping download of: 
> log/190be289.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,099 | INFO  | Slave skipping download of: 
> log/0642f848.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,411 | INFO  | Slave skipping download of: 
> log/0c85f06d.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 

[jira] [Commented] (AMQ-5618) Infinite loop in log replay with Replicated LevelDB

2016-09-20 Thread Pablo Lozano (JIRA)

[ 
https://issues.apache.org/jira/browse/AMQ-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508175#comment-15508175
 ] 

Pablo Lozano commented on AMQ-5618:
---

Actually no, and my team discarded the usage of Replicated Level DB
But on a certain hope, another team from my company started using Replicated 
Level DB and until now they haven't seen this issue occur.
They are running 5.13.3  and Java version of LevelDB. I think they have 
stumbled with other issues related to Zookeeper but not this one.

Important question, is this issue still occurring to you?

Thanks,
Pablo 

> Infinite loop in log replay with Replicated LevelDB
> ---
>
> Key: AMQ-5618
> URL: https://issues.apache.org/jira/browse/AMQ-5618
> Project: ActiveMQ
>  Issue Type: Bug
>Affects Versions: 5.11.0, 5.11.1
> Environment: Linux, Google Compute Engine
>Reporter: Artem Karpenko
>Priority: Critical
>
> This is very similar to AMQ-5300 except that I use replicatedLevelDB 
> persistence adapter and in order to reproduce I don't have to delete any 
> index files.
> Setup: 1 ZK instance, 3 AMQ nodes.
> One of the AMQ configs:
> {code}
>  replicas="3"
> bind="tcp://0.0.0.0:61619"
> zkAddress="instance-6:2181"
> zkPath="/activemq/leveldb-stores"
> hostname="instance-7" />
> {code}
> Difference between nodes is only in hostname attribute.
> The way to reproduce is almost the same as in AMQ-5300: 
> # Produce lots of messages to generate several log files in leveldb data 
> directory.
> # Consume _some_ messages until you see "Deleting log" in activemq.log.
> # Restart master. Wait for system to rebalance itself. Everything's fine at 
> this point.
> # Restart the second master.
> # Observe the massive (infinite?) logging on slave and relatively calm but 
> still possibly infinite logging on master.
> This is what the first master logs after it's restarted:
> {code}
> 2015-02-25 21:37:08,338 | DEBUG | Download session connected... | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:08,582 | INFO  | Slave skipping download of: 
> log/190be289.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,099 | INFO  | Slave skipping download of: 
> log/0642f848.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,411 | INFO  | Slave skipping download of: 
> log/0c85f06d.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,838 | INFO  | Slave skipping download of: 
> log/12c8e921.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,842 | INFO  | Slave requested: 
> 1c9373b4.index/CURRENT | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,846 | INFO  | Slave requested: 
> 1c9373b4.index/MANIFEST-02 | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,850 | INFO  | Slave requested: 
> 1c9373b4.index/03.log | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,857 | INFO  | Attaching... Downloaded 0.02/95.65 kb and 
> 1/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,859 | INFO  | Attaching... Downloaded 0.06/95.65 kb and 
> 2/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,861 | INFO  | Attaching... Downloaded 95.65/95.65 kb and 
> 3/3 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,862 | INFO  | Attached | 
> org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | 
> hawtdispatch-DEFAULT-1
> 2015-02-25 21:37:09,878 | DEBUG | Taking a snapshot of the current index: 
> /usr/local/apache-activemq-5.11.1/data/replicatedLevelDB/1c9373b4.index
>  | org.apache.activemq.leveldb.LevelDBClient | Thread-2
> 2015-02-25 21:37:10,352 | DEBUG | Recovering from last index snapshot at: 
> /usr/local/apache-activemq-5.11.1/data/replicatedLevelDB/dirty.index | 
> org.apache.activemq.leveldb.LevelDBClient | Thread-2
> {code}
> Right after that everything seems fine. But as soon as I stop the new master, 
> the another new master (that would be the third one) logs
> {code}
> 2015-02-25 21:38:43,876 | INFO  | Promoted to master | 
>