When a message is stuck on a given queue, have you confirmed that there is at least one consumer for that queue connected to the broker on which the message resides? A JMX viewer such as JConsole is ideal for determining this.
I haven't used the rebalanceClusterClients option before, but I expect that it's not able to guarantee that there is at least one consumer on each broker for each queue, so I could envision a situation where small numbers of messages get stuck on a given broker. If that turns out to be what's going on, you should probably use the replayWhenNoConsumers option described in the Stuck Messages section of http://activemq.apache.org/networks-of-brokers.html instead of (or in addition to, if you want) the rebalanceClusterClients option. Also, I don't understand why you have both rebalanceClusterClients=true and priorityBackup=true; those two options work at cross-purposes. What are you trying to accomplish with the combination of the two? Tim On Thu, Sep 14, 2017 at 5:46 PM, rockies <[email protected]> wrote: > Hi, > > We have setup network of brokers with just two ActiveMQ nodes in version > 5.15.0 > Config is as follows: > <networkConnectors> > <networkConnector conduitSubscriptions="false" > decreaseNetworkConsumerPriority="true" duplex="true" dynamicOnly="true" > name="broker1Tobroker02" networkTTL="3" userName="testadmin" > password="password" uri="static:tcp://xxxx.test.com:61616"> > </networkConnector> > </networkConnectors> > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage percentOfJvmHeap="70" /> > </memoryUsage> > <storeUsage> > <storeUsage limit="100 gb"/> > </storeUsage> > <tempUsage> > <tempUsage limit="50 gb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > > <transportConnectors> > > <transportConnector name="openwire" > rebalanceClusterClients="true" updateClusterClients="true" > updateClusterClientsOnRemove="true" > uri="tcp://0.0.0.0:61616?maximumConnections=1000& > wireFormat.maxFrameSize=104857600"/> > <transportConnector name="amqp" rebalanceClusterClients="true" > updateClusterClients="true" updateClusterClientsOnRemove="true" > uri="amqp://0.0.0.0:5672?maximumConnections=1000& > wireFormat.maxFrameSize=104857600"/> > <transportConnector name="stomp" > uri="stomp://0.0.0.0:61613?maximumConnections=1000& > wireFormat.maxFrameSize=104857600"/> > <transportConnector name="mqtt" > uri="mqtt://0.0.0.0:1883?maximumConnections=1000& > wireFormat.maxFrameSize=104857600"/> > <transportConnector name="ws" > uri="ws://0.0.0.0:61614?maximumConnections=1000& > wireFormat.maxFrameSize=104857600"/> > </transportConnectors> > > > Data is persisted in SQL Server. We use Mule ESB cluster, > where > subscribers from both nodes listen to Queues in ActiveMQ with a failover > URI > such as : > failover://(tcp://broker1.test.com:61616,tcp://broker2.test.com:61616 > )?randomize=false&priorityBackup=true&initialReconnectDelay=100 > > Since this updgrade from 5.11.1 to 5.14.5 to 5.15.0 we are > seem to have > more stuck messages in our queues, which usually ends up in re-starting JMS > brokers and ESB servers. There are atleast close to 200 queues. Message > size > rarely gets to 25MB and number of messages per second could be close to 50. > So why is it that we seem to have more issues with stuck messages ? Can you > please let us know what whould be the optimal configuration for this kind > of > clustered environment? > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >
