Justin, > Where exactly do you see the re-ordering of messages and how are you > determining that they are, in fact, re-ordered? Are they re-ordered in the > target queue itself or on the consumer(s)? My test-suite has a single-threaded amqp-producer that produces a few hundred-thousand sequentially numbered messages on the first node of a regular cluster; and there is a single threaded amqp-consumer that consumes these messages from the second node. There are no other nodes (I reduced the cluster to these 2 nodes). The consumer validates the sequence number. I currently have no tools/tests to inspect the messages while they are inside Artemis.
Note that due to the amount of messags, paging is in effect on both brokers. > [sample] Here is a sample from the log from my test-case: (all messages before this are properly delivered in sequence) WARNING: recv#1: expected message 'msg#3955', but got 'msg#3962' WARNING: recv#1: expected message 'msg#3956', but got 'msg#3955' WARNING: recv#1: expected message 'msg#3957', but got 'msg#3956' WARNING: recv#1: expected message 'msg#3958', but got 'msg#3957' WARNING: recv#1: expected message 'msg#3959', but got 'msg#3958' WARNING: recv#1: expected message 'msg#3960', but got 'msg#3959' WARNING: recv#1: expected message 'msg#3961', but got 'msg#3960' WARNING: recv#1: expected message 'msg#3962', but got 'msg#3961' (roughly the next 2000 messages are properly processed in sequence again) Message number 3962 appears too early by, and a few messages later it is in-sync again. The exact sequence number at which the effect appears is (wildly) different on each run. > Do you have a way that I can reproduce this behavior? My test-suite is too large to provide directly, and it also contains customer-specific scenarios. I'll spend some time to create a reproducer. But first, I'll also redo the test using a single node. Erwin -----Original Message----- From: Justin Bertram <jbert...@apache.org> Sent: Friday, May 3, 2024 8:57 PM To: users@activemq.apache.org Subject: Re: question on message-redistribution in an Artemis cluster EXTERNAL SENDER: Do not click any links or open any attachments unless you trust the sender and know the content is safe. EXPÉDITEUR EXTERNE: Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe à moins qu’ils ne proviennent d’un expéditeur fiable, ou que vous ayez l'assurance que le contenu provient d'une source sûre. Where exactly do you see the re-ordering of messages and how are you determining that they are, in fact, re-ordered? Are they re-ordered in the target queue itself or on the consumer(s)? Do you have a way that I can reproduce this behavior? If I recall correctly there is just one thread that forwards messages from the internal store-and-forward queue to the target node so I don't think that would be the reason for this behavior. Justin On Fri, May 3, 2024 at 10:02 AM Dondorp, Erwin <erwin.dond...@ns.nl.invalid> wrote: > Intern > > Hello, > > I have an Artemis cluster with several nodes. > In many cases, messages arrive on one node and are consumed from > another node. > The cluster-connections between the nodes do their work to make that > happen. > > But under high-load we see messages being re-ordered between the > broker-nodes, and I am trying to minimize (or eliminate) that effect. > The re-ordering is usually only within a small set of messages, e.g. > within a set of 10 to 20 messages. > My assumption is that this is due to the use of multiple threads in > the cluster-connections when there are a lot of messages. > Is there a way to have more control over this? > > thx! > Erwin > > ________________________________ > > Deze e-mail, inclusief eventuele bijlagen, is uitsluitend bestemd voor > (gebruik door) de geadresseerde. De e-mail kan persoonlijke of > vertrouwelijke informatie bevatten. Openbaarmaking, vermenigvuldiging, > verspreiding en/of verstrekking van (de inhoud van) deze e-mail (en > eventuele bijlagen) aan derden is uitdrukkelijk niet toegestaan. > Indien u niet de bedoelde geadresseerde bent, wordt u vriendelijk > verzocht degene die de e-mail verzond hiervan direct op de hoogte te > brengen en de e-mail (en eventuele bijlagen) te vernietigen. > > Informatie > vennootschap<https://urldefense.com/v3/__http://www.ns.nl/emaildisclai > mer__;!!AaIhyw!psLFP82x4zK7ZZ2qL1XdXWq58ZnIo5ZhoCIFSQ0P34fxf7Ik5pR9mjB > IiZm5FSjzVGxGj1gD0wBPk-uRngk$ > > > > Intern >