Can you elaborate on the use-case a bit? What clients are connected to which nodes?
Also, what's the minimal environment you need to reproduce the issue? It looks like maybe you could reproduce this with just 2 live brokers in a cluster and using the Artemis CLI to send & receive messages, but without more details about your use-case it's hard to tell. An automated reproducer or at least steps to reproduce would be great. You need to be specifically invited to the Apache slack in order to join. Is the email address you sent the original message from the one you'd want to use to receive the invite? Justin On Tue, Sep 15, 2020 at 9:23 AM Paul Whelan <paul.whe...@sentenial.com> wrote: > Technical Details > > Machine 1 mom1 > Master 1 IP 10.70.120.243 PORT 61616 > Slave 2 IP 10.70.120.243 PORT 62626 > Machine 2 mom2 > Master 2 IP 10.70.120.221 PORT 61616 > Slave 1 IP 10.70.120.221 PORT 62626 > > Machine 3 Jenkins Slave > IP 10.74.175.58 > > Spring framework application > Using spring boot version 2.3.0.RELEASE > An instance of a master and slave are on each install > Packaged as a docker container. > Slave on machine 1 is paired with master on machine 2 > Slave on machine 2 is paired with master on machine 1 > > Using gatling to performance test the install we noticed missing messages. > Connection string (tcp://mom01:61616,tcp:// > mom02:61616)?ha=true&reconnectAttempts=-1 > > To aid with debugging restricted the test to just sending in one message, > here are the logs associated with this trial run. > Both master nodes are operational. > > # msg1 (correct) > # node 1 mom1 > 2020-09-14 20:45:27.864 [386] TRACE o.a.a.a.c.s.i.ServerSessionImpl - > send(message=CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=497,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]]@861485015, > direct=true) being called > 2020-09-14 20:45:27.865 [386] TRACE o.a.a.a.c.s.c.i.RemoteQueueBindingImpl > - Adding remoteQueue ID = 2044128 into > message=CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 > store-forward-queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 > 2020-09-14 20:45:27.866 [386] INFO > c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2781115, > address: GATLING_JMS_TEST_IN, correlationId: > 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: > [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h > e l l o ]}, session name: d6ea028b-f6c2-11ea-a9da-0050569d3ae9, session > connection: [clientId: null, remoteAddress: /10.74.175.58:41692], result: > OK > 2020-09-14 20:45:27.873 [13922] DEBUG o.a.a.a.c.s.i.QueueImpl - > QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 doing deliver. messageReferences=0 > 2020-09-14 20:45:27.874 [13922] TRACE o.a.a.a.c.s.i.QueueImpl - Queue > $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895 > is delivering reference > Reference[2781115]:RELIABLE:CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 > 2020-09-14 20:45:27.880 [13922] TRACE o.a.a.a.c.s.c.i.BridgeImpl - Bridge > ClusterConnectionBridge@3e3abc7e > [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 targetConnector=ServerLocatorImpl > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3e3abc7e > [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 targetConnector=ServerLocatorImpl > [initialConnectors=[TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom02-sent-local], > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@772166315[nodeUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55, > connector=TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom01-sent-local, address=, > server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55])) > [initialConnectors=[TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]] is > handling > reference=Reference[2781115]:RELIABLE:CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 > > Messages seem to have arrived into node1 mom1 and they are being > transported over to node2 mom2 > > # node 2 mom2 > 2020-09-14 20:45:27.890 [352] INFO > c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2169502, > address: GATLING_JMS_TEST_IN, correlationId: > 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: > [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]], > body: [ h e l l o ]}, session name: > 5644a321-f449-11ea-9ce1-0242b5cd1d55, session connection: [clientId: null, > remoteAddress: /10.70.120.243:37768], result: OK > 2020-09-14 20:45:27.893 [324] INFO > c.s.a.s.SL4JLoggingActiveMQServerPlugin - delivered message [ID: 2169502, > address: GATLING_JMS_TEST_IN, correlationId: > 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: > [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]], > body: [ h e l l o ], to consumer [connectionClientId: null, > remoteAddres: /10.74.175.58:59048, queueName: GATLING_JMS_TEST_IN, > sessionName: d6a9c537-f6c2-11ea-a9da-0050569d3ae9] > # msg2 (NO_BINDINGS) > 2020-09-14 20:45:27.969 [405] TRACE o.a.a.a.c.s.i.ServerSessionImpl - > send(message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=497,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,JMSType=test_jms_type]]@660678797, > direct=true) being called > 2020-09-14 20:45:27.969 [405] TRACE o.a.a.a.c.s.c.i.RemoteQueueBindingImpl > - Adding remoteQueue ID = 2044128 into > message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797 > store-forward-queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 > 2020-09-14 20:45:27.971 [405] INFO > c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2781126, > address: > $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > correlationId: d48d23aa-dfb7-46da-948a-fa387ea61b41, props: > [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h > e l l o ]}, session name: d701a93f-f6c2-11ea-a9da-0050569d3ae9, session > connection: [clientId: null, remoteAddress: /10.74.175.58:41692], result: > OK > 2020-09-14 20:45:27.975 [13901] DEBUG o.a.a.a.c.s.i.QueueImpl - > QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 doing deliver. messageReferences=0 > 2020-09-14 20:45:27.975 [13901] TRACE o.a.a.a.c.s.i.QueueImpl - Queue > $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895 > is delivering reference PagedReferenceImpl [position=PagePositionImpl > [pageNr=2818, messageNr=0, recordID=-1, fileOffset=0], > message=PagedMessageImpl [queueIDs=[4403], transactionID=-1, > message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895,size=798,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797], > deliveryTime=0, persistedCount=0, deliveryCount=0, > subscription=PageSubscriptionImpl [cursorId=4403, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408, filter = null]] > 2020-09-14 20:45:27.975 [13901] TRACE o.a.a.a.c.s.c.i.BridgeImpl - Bridge > ClusterConnectionBridge@3e3abc7e > [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 targetConnector=ServerLocatorImpl > (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3e3abc7e > [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408 targetConnector=ServerLocatorImpl > [initialConnectors=[TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom02-sent-local], > discoveryGroupConfiguration=null]]::ClusterConnectionImpl@772166315[nodeUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55, > connector=TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom01-sent-local, address=, > server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55])) > [initialConnectors=[TransportConfiguration(name=netty-connector, > factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) > ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]] is > handling reference=PagedReferenceImpl [position=PagePositionImpl > [pageNr=2818, messageNr=0, recordID=-1, fileOffset=0], > message=PagedMessageImpl [queueIDs=[4403], transactionID=-1, > message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, > timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, > address=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895,size=798,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797], > deliveryTime=0, persistedCount=0, deliveryCount=0, > subscription=PageSubscriptionImpl [cursorId=4403, > queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > postOffice=PostOfficeImpl > [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], > temp=false]@32961408, filter = null]] > 2020-09-14 20:45:27.980 [378] INFO > c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2169509, > address: > $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, > correlationId: d48d23aa-dfb7-46da-948a-fa387ea61b41, props: > [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO=[0000 > 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h > e l l o ]}, session name: 5644a321-f449-11ea-9ce1-0242b5cd1d55, session > connection: [clientId: null, remoteAddress: /10.70.120.243:37768], > result: NO_BINDINGS > > The message goes missing due to the NO_BINDINGS. > How to resolve this issue? > > > PS: How do I sign up for the slack channel without an apache.org email > address? > Thanks > Paul > The views and opinions expressed in this email may not reflect the views > and opinions of any member of Sentenial Limited. The information contained > in this message is confidential and may also be privileged. It is intended > only for the addressee named above. The unauthorised use, disclosure, > copying or alteration of this message is strictly prohibited. If you are > not the addressee (or responsible for delivery of the message to the > addressee), please notify the originator immediately by return message and > destroy the original message. This message and any attachments have been > scanned for viruses prior to leaving our network. However, we do not > guarantee the security of this message and will not be responsible for any > damages arising as a result of any virus being passed on or arising from > any alteration of this message by a third party. We may monitor emails sent > to and from our network. Sentenial Limited; Registered in Ireland, No. > 374137. Registered Office: Unit 16F, Maynooth Business Campus, Maynooth, Co > Kildare, Ireland. >