Technical Details Machine 1 mom1 Master 1 IP 10.70.120.243 PORT 61616 Slave 2 IP 10.70.120.243 PORT 62626 Machine 2 mom2 Master 2 IP 10.70.120.221 PORT 61616 Slave 1 IP 10.70.120.221 PORT 62626
Machine 3 Jenkins Slave IP 10.74.175.58 Spring framework application Using spring boot version 2.3.0.RELEASE An instance of a master and slave are on each install Packaged as a docker container. Slave on machine 1 is paired with master on machine 2 Slave on machine 2 is paired with master on machine 1 Using gatling to performance test the install we noticed missing messages. Connection string (tcp://mom01:61616,tcp:// mom02:61616)?ha=true&reconnectAttempts=-1 To aid with debugging restricted the test to just sending in one message, here are the logs associated with this trial run. Both master nodes are operational. # msg1 (correct) # node 1 mom1 2020-09-14 20:45:27.864 [386] TRACE o.a.a.a.c.s.i.ServerSessionImpl - send(message=CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=497,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]]@861485015, direct=true) being called 2020-09-14 20:45:27.865 [386] TRACE o.a.a.a.c.s.c.i.RemoteQueueBindingImpl - Adding remoteQueue ID = 2044128 into message=CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 store-forward-queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 2020-09-14 20:45:27.866 [386] INFO c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2781115, address: GATLING_JMS_TEST_IN, correlationId: 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h e l l o ]}, session name: d6ea028b-f6c2-11ea-a9da-0050569d3ae9, session connection: [clientId: null, remoteAddress: /10.74.175.58:41692], result: OK 2020-09-14 20:45:27.873 [13922] DEBUG o.a.a.a.c.s.i.QueueImpl - QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 doing deliver. messageReferences=0 2020-09-14 20:45:27.874 [13922] TRACE o.a.a.a.c.s.i.QueueImpl - Queue $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895 is delivering reference Reference[2781115]:RELIABLE:CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 2020-09-14 20:45:27.880 [13922] TRACE o.a.a.a.c.s.c.i.BridgeImpl - Bridge ClusterConnectionBridge@3e3abc7e [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3e3abc7e [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@772166315[nodeUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom01-sent-local, address=, server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]] is handling reference=Reference[2781115]:RELIABLE:CoreMessage[messageID=2781115,durable=true,userID=d6f3033c-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@861485015 Messages seem to have arrived into node1 mom1 and they are being transported over to node2 mom2 # node 2 mom2 2020-09-14 20:45:27.890 [352] INFO c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2169502, address: GATLING_JMS_TEST_IN, correlationId: 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]], body: [ h e l l o ]}, session name: 5644a321-f449-11ea-9ce1-0242b5cd1d55, session connection: [clientId: null, remoteAddress: /10.70.120.243:37768], result: OK 2020-09-14 20:45:27.893 [324] INFO c.s.a.s.SL4JLoggingActiveMQServerPlugin - delivered message [ID: 2169502, address: GATLING_JMS_TEST_IN, correlationId: 6c54e907-d5d6-492c-8a9c-9b6789948b7d, props: [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=6c54e907-d5d6-492c-8a9c-9b6789948b7d,JMSType=test_jms_type]], body: [ h e l l o ], to consumer [connectionClientId: null, remoteAddres: /10.74.175.58:59048, queueName: GATLING_JMS_TEST_IN, sessionName: d6a9c537-f6c2-11ea-a9da-0050569d3ae9] # msg2 (NO_BINDINGS) 2020-09-14 20:45:27.969 [405] TRACE o.a.a.a.c.s.i.ServerSessionImpl - send(message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=497,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,JMSType=test_jms_type]]@660678797, direct=true) being called 2020-09-14 20:45:27.969 [405] TRACE o.a.a.a.c.s.c.i.RemoteQueueBindingImpl - Adding remoteQueue ID = 2044128 into message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=GATLING_JMS_TEST_IN,size=688,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797 store-forward-queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 2020-09-14 20:45:27.971 [405] INFO c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2781126, address: $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, correlationId: d48d23aa-dfb7-46da-948a-fa387ea61b41, props: [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h e l l o ]}, session name: d701a93f-f6c2-11ea-a9da-0050569d3ae9, session connection: [clientId: null, remoteAddress: /10.74.175.58:41692], result: OK 2020-09-14 20:45:27.975 [13901] DEBUG o.a.a.a.c.s.i.QueueImpl - QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 doing deliver. messageReferences=0 2020-09-14 20:45:27.975 [13901] TRACE o.a.a.a.c.s.i.QueueImpl - Queue $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895 is delivering reference PagedReferenceImpl [position=PagePositionImpl [pageNr=2818, messageNr=0, recordID=-1, fileOffset=0], message=PagedMessageImpl [queueIDs=[4403], transactionID=-1, message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895,size=798,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797], deliveryTime=0, persistedCount=0, deliveryCount=0, subscription=PageSubscriptionImpl [cursorId=4403, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408, filter = null]] 2020-09-14 20:45:27.975 [13901] TRACE o.a.a.a.c.s.c.i.BridgeImpl - Bridge ClusterConnectionBridge@3e3abc7e [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@3e3abc7e [name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@772166315[nodeUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55, connector=TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom01-sent-local, address=, server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55])) [initialConnectors=[TransportConfiguration(name=netty-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=mom02-sent-local], discoveryGroupConfiguration=null]] is handling reference=PagedReferenceImpl [position=PagePositionImpl [pageNr=2818, messageNr=0, recordID=-1, fileOffset=0], message=PagedMessageImpl [queueIDs=[4403], transactionID=-1, message=CoreMessage[messageID=2781126,durable=true,userID=d702e1c0-f6c2-11ea-a9da-0050569d3ae9,priority=4, timestamp=Mon Sep 14 20:45:27 IST 2020,expiration=0, durable=true, address=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895,size=798,properties=TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]]@660678797], deliveryTime=0, persistedCount=0, deliveryCount=0, subscription=PageSubscriptionImpl [cursorId=4403, queue=QueueImpl[name=$.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=89ec03ba-f448-11ea-9ce1-0242b5cd1d55], temp=false]@32961408, filter = null]] 2020-09-14 20:45:27.980 [378] INFO c.s.a.s.SL4JLoggingActiveMQServerPlugin - sent message [ID: 2169509, address: $.artemis.internal.sf.artemis-cluster.50f5d790-f449-11ea-a593-0242ed448895, correlationId: d48d23aa-dfb7-46da-948a-fa387ea61b41, props: [TypedProperties[__AMQ_CID=d6d51af8-f6c2-11ea-a9da-0050569d3ae9,JMSReplyTo=queue://GATLING_JMS_TEST_OUT,_AMQ_ROUTING_TYPE=1,JMSCorrelationID=d48d23aa-dfb7-46da-948a-fa387ea61b41,_AMQ_ROUTE_TO=[0000 0000 001F 30E0),bytesAsLongs(2044128],JMSType=test_jms_type]], body: [ h e l l o ]}, session name: 5644a321-f449-11ea-9ce1-0242b5cd1d55, session connection: [clientId: null, remoteAddress: /10.70.120.243:37768], result: NO_BINDINGS The message goes missing due to the NO_BINDINGS. How to resolve this issue? PS: How do I sign up for the slack channel without an apache.org email address? Thanks Paul The views and opinions expressed in this email may not reflect the views and opinions of any member of Sentenial Limited. The information contained in this message is confidential and may also be privileged. It is intended only for the addressee named above. The unauthorised use, disclosure, copying or alteration of this message is strictly prohibited. If you are not the addressee (or responsible for delivery of the message to the addressee), please notify the originator immediately by return message and destroy the original message. This message and any attachments have been scanned for viruses prior to leaving our network. However, we do not guarantee the security of this message and will not be responsible for any damages arising as a result of any virus being passed on or arising from any alteration of this message by a third party. We may monitor emails sent to and from our network. Sentenial Limited; Registered in Ireland, No. 374137. Registered Office: Unit 16F, Maynooth Business Campus, Maynooth, Co Kildare, Ireland.
