[jira] [Commented] (ARTEMIS-1864) On-Demand Message Redistribution Can Spontaneously Start Failing in Single Direction
[
https://issues.apache.org/jira/browse/ARTEMIS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705785#comment-16705785
]
Alexander Kosarev commented on ARTEMIS-1864:
I created an issue in Spring's Jira [https://jira.spring.io/browse/SPR-17554.]
Example project and configuration can be found there.
> On-Demand Message Redistribution Can Spontaneously Start Failing in Single
> Direction
>
>
> Key: ARTEMIS-1864
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1864
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.5.0
> Environment: RHEL 6.2
>Reporter: Ilkka Virolainen
>Priority: Major
>
> It's possible that the message redistribution of an Artemis cluster can
> spontaneously fail after running a while. I've witnessed this several times
> using a two node colocated replicating cluster with a basic configuration:
> {code:java}
>
>
> netty-connector
> 500
> 5
> true
> ON_DEMAND
> 1
>
>
> {code}
> After running a while (approx. two weeks) one of the nodes (node a) will stop
> consuming messages from the other node's (node b) internal store-and-forward
> queue. This will result in message redistribution not working from node b ->
> node a but will work from node a -> node b. The cause for this is unknown:
> nothing of note is logged for either broker and JMX shows that the cluster
> topology and the broker cluster bridge connection are intact. This will cause
> significant problems, mainly:
> 1. Client communication will only work as expected if the clients happen to
> connect to the right brokers
> 2. Unconsumed messages will end up piling in the internal store-and-forward
> queue and consume unnecessary resources. It's also possible (but not
> verified) that when messages in the internal queue expire, they leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1864) On-Demand Message Redistribution Can Spontaneously Start Failing in Single Direction
[
https://issues.apache.org/jira/browse/ARTEMIS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704221#comment-16704221
]
Alexander Kosarev commented on ARTEMIS-1864:
[~jbertram] I found source of the issue in my case - I used Spring Framework
JmsTemplate with default configuration. Spring JMS closes connection with a
message broker after every message sent, and it causes described problems with
a cluster (my bad that I didn't notice a note about avoiding anti-patterns in
the
[documentation|https://activemq.apache.org/artemis/docs/latest/perf-tuning.html]).
So my solution is to use *PooledConnectionFactory*.
I can provide an example Spring-based application that can cause problems with
Artemis cluster.
> On-Demand Message Redistribution Can Spontaneously Start Failing in Single
> Direction
>
>
> Key: ARTEMIS-1864
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1864
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.5.0
> Environment: RHEL 6.2
>Reporter: Ilkka Virolainen
>Priority: Major
>
> It's possible that the message redistribution of an Artemis cluster can
> spontaneously fail after running a while. I've witnessed this several times
> using a two node colocated replicating cluster with a basic configuration:
> {code:java}
>
>
> netty-connector
> 500
> 5
> true
> ON_DEMAND
> 1
>
>
> {code}
> After running a while (approx. two weeks) one of the nodes (node a) will stop
> consuming messages from the other node's (node b) internal store-and-forward
> queue. This will result in message redistribution not working from node b ->
> node a but will work from node a -> node b. The cause for this is unknown:
> nothing of note is logged for either broker and JMX shows that the cluster
> topology and the broker cluster bridge connection are intact. This will cause
> significant problems, mainly:
> 1. Client communication will only work as expected if the clients happen to
> connect to the right brokers
> 2. Unconsumed messages will end up piling in the internal store-and-forward
> queue and consume unnecessary resources. It's also possible (but not
> verified) that when messages in the internal queue expire, they leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1864) On-Demand Message Redistribution Can Spontaneously Start Failing in Single Direction
[
https://issues.apache.org/jira/browse/ARTEMIS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699590#comment-16699590
]
Justin Bertram commented on ARTEMIS-1864:
-
Can you provide clear steps to reproduce?
> On-Demand Message Redistribution Can Spontaneously Start Failing in Single
> Direction
>
>
> Key: ARTEMIS-1864
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1864
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.5.0
> Environment: RHEL 6.2
>Reporter: Ilkka Virolainen
>Priority: Major
>
> It's possible that the message redistribution of an Artemis cluster can
> spontaneously fail after running a while. I've witnessed this several times
> using a two node colocated replicating cluster with a basic configuration:
> {code:java}
>
>
> netty-connector
> 500
> 5
> true
> ON_DEMAND
> 1
>
>
> {code}
> After running a while (approx. two weeks) one of the nodes (node a) will stop
> consuming messages from the other node's (node b) internal store-and-forward
> queue. This will result in message redistribution not working from node b ->
> node a but will work from node a -> node b. The cause for this is unknown:
> nothing of note is logged for either broker and JMX shows that the cluster
> topology and the broker cluster bridge connection are intact. This will cause
> significant problems, mainly:
> 1. Client communication will only work as expected if the clients happen to
> connect to the right brokers
> 2. Unconsumed messages will end up piling in the internal store-and-forward
> queue and consume unnecessary resources. It's also possible (but not
> verified) that when messages in the internal queue expire, they leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1864) On-Demand Message Redistribution Can Spontaneously Start Failing in Single Direction
[
https://issues.apache.org/jira/browse/ARTEMIS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696622#comment-16696622
]
Alexander Kosarev commented on ARTEMIS-1864:
We have the same issue with both Apache Artemis 2.6.3 and JBoss MQ 7 (build
2.6.3.redhat-4).
Cluster configuration:
2 nodes with configuration:
{code:java}
tcp://192.167.1.10:61616
admin
admin
${udp-address:231.7.7.7}
9876
100
node02-connector
${udp-address:231.7.7.7}
9876
1
node02-connector
true
1
0
{code}
There are multiple ActiveMQ 5.15.6 JMS clients configured with
failover-transport:
{code:java}
failover://(tcp://host1:port,tcp://host2:port){code}
All clients can consume and produce messages.
Messages stick in queue
*$.artemis.internal.sf.sandbox-cluster.CLUSTER_NAME.NODE_NAME* at some point in
"Delivering" state with following exception:
{code:java}
2018-11-23 14:36:25,274 WARN [org.apache.activemq.artemis.core.server]
AMQ222151: removing consumer which did not handle a message,
consumer=ClusterConnectionBridge@299a7489
[name=$.artemis.internal.sf.sandbox-cluster.e0702bcf-e636-11e8-bca1-6aabda98944e,
queue=QueueImpl
[name=$.artemis.internal.sf.sandbox-cluster.e0702bcf-e636-11e8-bca1-6aabda98944e,
postOffice=PostOfficeImpl
[server=ActiveMQServerImpl::serverUUID=b42fe4db-e636-11e8-b335-6aabda98944e],
temp=false]@1584523b targetConnector=ServerLocatorImpl
(identity=(Cluster-connection-b
ridge::ClusterConnectionBridge@299a7489
[name=$.artemis.internal.sf.sandbox-cluster.e0702bcf-e636-11e8-bca1-6aabda98944e,
queue=QueueImpl[name=$.artemis.internal.sf.sandbox-cluster.e0702bcf-e636-11e8-bca1-6aabda98944e,
postOffice=PostOfficeImpl [server=ActiveMQServerImpl:
:serverUUID=b42fe4db-e636-11e8-b335-6aabda98944e], temp=false]@1584523b
targetConnector=ServerLocatorImpl
[initialConnectors=[TransportConfiguration(name=node02-connector,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61716
=10-145-13-120],
discoveryGroupConfiguration=null]]::ClusterConnectionImpl@638169719[nodeUUID=b42fe4db-e636-11e8-b335-6aabda98944e,
connector=TransportConfiguration(name=node01-connector,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?port=61616=10-145-13-120, address=,
server=ActiveMQServerImpl::serverUUID=b42fe4db-e636-11e8-b335-6aabda98944e]))
[initialConnectors=[TransportConfiguration(name=node02-connector,
factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory)
?p
ort=61716=10-145-13-120], discoveryGroupConfiguration=null]],
message=Reference[2151226563]:NON-RELIABLE:CoreMessage[messageID=2151226563,durable=false,userID=null,priority=0,
timestamp=0,expiration=0, durable=false,
address=ActiveMQ.Advisory.TempQueue,size=1077,prop
erties=TypedProperties[__HDR_BROKER_IN_TIME=1542965785270,_AMQ_ROUTING_TYPE=0,__HDR_GROUP_SEQUENCE=0,__HDR_COMMAND_ID=0,__HDR_DATASTRUCTURE=[
0062 0800 0178 0100 2449 443A 616B 6F73 6172 6576 2D33 3933 ... 3535
2D62 6236 352D 3966 3165 6361 3033 3861 3766
0100
),_AMQ_DUPL_ID=ID:akosarev-46097-1542964149858-1:1:0:0:21605,__HDR_MESSAGE_ID=[
004A 6E00 017B 0100 2349 443A 616B 6F73 6172 6576 2D34 3630 3937 2D31 ...
5465
),__HDR_DROPPA
BLE=false,__HDR_ARRIVAL=0,_AMQ_ROUTE_TO$.artemis.internal.sf.sandbox-cluster.e0702bcf-e636-11e8-bca1-6aabda98944e=[
0035 C0C3),bytesAsLongs(3522755],__HDR_PRODUCER_ID=[ 0037 7B01 0023
4944 3A61 6B6F 7361 7265 762D 3436 3039 372D 3135 3432 3936 3431 3439 3835
382D 313A 3100
00),JMSType=Advisory]]@1531727971: java.lang.IndexOutOfBoundsException:
writerIndex: 4 (expected: readerIndex(0) <= writerIndex <= capacity(0))
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:118)
[netty-all-4.1.25.Final-redhat-3.jar:4.1.25.Final-redhat-3]
at io.netty.buffer.WrappedByteBuf.writerIndex(WrappedByteBuf.java:129)
[netty-all-4.1.25.Final-redhat-3.jar:4.1.25.Final-redhat-3]
at
org.apache.activemq.artemis.core.buffers.impl.ResetLimitWrappedActiveMQBuffer.writerIndex(ResetLimitWrappedActiveMQBuffer.java:128)
[artemis-core-client-2.6.3.redhat-4.jar:2.6.3.redhat-4]
at
org.apache.activemq.artemis.core.buffers.impl.ResetLimitWrappedActiveMQBuffer.(ResetLimitWrappedActiveMQBuffer.java:60)
[artemis-core-client-2.6.3.redhat-4.jar:2.6.3.redhat-4]
at
org.apache.activemq.artemis.core.message.impl.CoreMessage.internalWritableBuffer(CoreMessage.java:367)
[artemis-core-client-2.6.3.redhat-4.jar:2.6.3.redhat-4]
at
