[
https://issues.apache.org/jira/browse/ARTEMIS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883311#comment-17883311
]
Justin Bertram edited comment on ARTEMIS-5042 at 9/20/24 8:12 PM:
------------------------------------------------------------------
I believe the problem is being caused by this exception which is being thrown
and short-circuiting the removal of the cluster bridge flow record (resulting
in messages still being routed to the bridge's store-and-forward queue):
{noformat}
2024-09-20 10:34:50,998 WARN [org.apache.activemq.artemis.core.server]
AMQ222268: Failed to remove a record
java.lang.IllegalStateException: Cannot find binding
mtmccon_ucpmula_sms_193#mcla_simusmsc3_ucpmula_queue9eb3fb16-71d6-11ef-9298-506b8daae603
at
org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.removeBindingInternal(SimpleAddressManager.java:228)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.lambda$removeBinding$2(WildcardAddressManager.java:129)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitValues(AddressPartNode.java:271)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitNonWildcard(AddressPartNode.java:219)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressMap.visitMatching(AddressMap.java:64)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.removeBinding(WildcardAddressManager.java:129)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:988)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.removeBinding(ClusterConnectionImpl.java:1384)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.clearBindings(ClusterConnectionImpl.java:1249)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.close(ClusterConnectionImpl.java:1052)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.removeRecord(ClusterConnectionImpl.java:1619)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.fail(ClusterConnectionBridge.java:436)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:699)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:878)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:804)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:576)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1417)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:209)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1182)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
~[artemis-commons-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
~[artemis-commons-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
~[artemis-commons-2.35.0.jar:2.35.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[?:?]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.35.0.jar:2.35.0]{noformat}
I think this logic could be improved to simply log when a binding can't be
found in this context. I'll send a PR for this.
That said, this is not normal behavior, and I haven't been able to reproduce
this in my local tests.
was (Author: jbertram):
The problem is being caused by this exception which is being thrown and
short-circuiting the removal of the cluster bridge flow record (resulting in
messages still being routed to the bridge's store-and-forward queue):
{noformat}
2024-09-20 10:34:50,998 WARN [org.apache.activemq.artemis.core.server]
AMQ222268: Failed to remove a record
java.lang.IllegalStateException: Cannot find binding
mtmccon_ucpmula_sms_193#mcla_simusmsc3_ucpmula_queue9eb3fb16-71d6-11ef-9298-506b8daae603
at
org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.removeBindingInternal(SimpleAddressManager.java:228)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.lambda$removeBinding$2(WildcardAddressManager.java:129)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitValues(AddressPartNode.java:271)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressPartNode.visitNonWildcard(AddressPartNode.java:219)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.AddressMap.visitMatching(AddressMap.java:64)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.WildcardAddressManager.removeBinding(WildcardAddressManager.java:129)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:988)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.removeBinding(ClusterConnectionImpl.java:1384)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.clearBindings(ClusterConnectionImpl.java:1249)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.close(ClusterConnectionImpl.java:1052)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.removeRecord(ClusterConnectionImpl.java:1619)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.fail(ClusterConnectionBridge.java:436)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:699)
~[artemis-server-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:878)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:804)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:576)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1417)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:98)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:209)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1182)
~[artemis-core-client-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
~[artemis-commons-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
~[artemis-commons-2.35.0.jar:2.35.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
~[artemis-commons-2.35.0.jar:2.35.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[?:?]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.35.0.jar:2.35.0]{noformat}
I think this logic could be improved to simply log when a binding can't be
found in this context. I'll send a PR for this.
That said, this is not normal behavior, and I haven't been able to reproduce
this in my local tests.
> The load balancing is not working correctly when several brokers are down
> -------------------------------------------------------------------------
>
> Key: ARTEMIS-5042
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5042
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Balancer
> Affects Versions: 2.35.0
> Environment: Cluster Configuration broker.xml:
>
> <cluster-connections>
> <cluster-connection name="mccon">
> <connection-ttl>60000</connection-ttl>
> <call-timeout>5000</call-timeout>
> <retry-interval>500</retry-interval>
> <retry-interval-multiplier>1.0</retry-interval-multiplier>
> <max-retry-interval>5000</max-retry-interval>
> <initial-connect-attempts>-1</initial-connect-attempts>
> <reconnect-attempts>-1</reconnect-attempts>
> <forward-when-no-consumers>false</forward-when-no-consumers>
> <connector-ref>artemis</connector-ref>
> <message-load-balancing>ON_DEMAND</message-load-balancing>
> <max-hops>1</max-hops>
> <static-connectors>
> <connector-ref>node0</connector-ref>
> <connector-ref>node1</connector-ref>
> <connector-ref>node2</connector-ref>
>
> </static-connectors>
> </cluster-connection>
> </cluster-connections>
> Reporter: Juanjo Marin
> Assignee: Justin Bertram
> Priority: Major
> Attachments: 1NodoUp.PNG, 1nodeUp.PNG, 3NodesUp.PNG, 3Nodosup.PNG,
> ArtemisInternalWithMessage.PNG, EmptysQueue.PNG,
> QueueInternWithMessageNoConsumer.PNG, artemis.log, broker.xml
>
>
> The load balancing is not working correctly when several brokers are down.
> With the configuration we have in the cluster, we understand that this should
> not be happening. The load balancing should be smart and not send messages to
> the nodes that are down."
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
