Hello, We tested with LinkCapacity equal to 1 on the "normal" listener with debug level trace+, here are our findings: Our Cluster: [image: Diagram.jpg] We are using the broker-j, the consumer are connected to the dispatch routers before we start sending.
For use case 1: 1- the producer sends a message. 2- Consumer 1 issues one credit, receives the message without acknowledging. 3- the producer sends another message. 4- Consumer 2 in auto-ack mode issues one credit and receives the message. 5- we repeated steps 3 and 4 ten times. 6- Consumer 1 acknowledges. The results were correct: all the messages were correctly distributed to the idle consumer. For use case 2: 1- the producer send 10 messages while no credits were issued yet by the consumers. 2- Consumer 1 issues one credit, receives a message without acknowledging. 3- Consumer 2 in auto-ack mode issues one credit and times out after 5 seconds if nothing is received. 4- we repeated step 3 eight times. 5- Consumer 1 acknowledges. The results were not as expected: 4 messages were blocked in the outbound queue of the consumer 1 and consumer 2 was able to receive only 5 messages. We analysed the traces to follow the messages. We found that 4 messages were blocked in the dispatch 1. Conclusion: if no consumers are issuing credits (are busy) then the incoming messages will be pre-assigned automatically by the dispatch router to the listeners (in a round robin way?). Is it an expected behavior in the dispatch router? is it not supposed to wait for a credit to be issued before binding the message to an outbound queue? If you want to check the paths of the messages, I attached the routers logs of the use case 2. Best regards, Rabih On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[email protected]> wrote: > On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[email protected]> wrote: > > > Hello, > > > > We are testing with the trace+ log. We will brief you of the results when > > they are ready. > > > > Our goal is not to load balance equally the messages between the > consumers > > but we would like the dispatch router to send the message to the free > > consumer. > > > > What you did (setting the linkCapacity on the producer and consumer > listeners to 1) is the right thing to do if you want the router > to send the message to *any* free consumer. > > If Consumer C1 is processing the first message and does not issue the next > credit until it finishes processing the first message and if the second > message arrives to Router 1 when C1 is > still processing the message, then Router 1 will definitely forward the > message to Router 2. BUT if C1 is fast enough and ends up processing the > first message and immediately issues credit, the second message if it > arrives in Router 1 will also > be sent to C1 (because the router prefers local consumers). > > Remember that the key here is which Router ends up getting the message > since each broker has two autoLinks to both routers and we don't know which > autoLink the broker will choose. > > But overall, with your new configuration, the router will send the message > to a consumer that is not busy no further configuration is necessary. > > Specially if the consumer does a long calculation, we do not want to block > > the message in the outbound queue knowing there is another idle consumer. > > > > Is there any special configuration for this? > > > > Best regards, > > Rabih > > > > > > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[email protected]> > wrote: > > > > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[email protected]> > > wrote: > > > > > > > Hello, > > > > > > > > Thank you Ganesh for your answer. > > > > To answer your first question yes the consumers we have, issue one > > credit > > > > per call. > > > > For the the second question, yes we are attaching consumers before > > > sending > > > > the messages. > > > > > > > > After setting the dispatch "link capacity" on the listeners to 1, we > > > > managed to have an equal load-balancing of the messages between > > consumer > > > 1 > > > > and 2. > > > > Our understanding is that the problem comes from the prefetch at the > > > level > > > > of the "normal" listener, which is preventing the "inter-router" > > listener > > > > to pass messages to the second dispatch. > > > > > > > Just want to note here that the linkCapacity on the inter-router > > listeners > > > are ignored. So setting linkCapacity to 1 on that listener has no > effect. > > > > > > This is what I think is happening > > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them > > connect > > > to both Brokers, B1 and B2 and establish the autolinks. As soon as > > inbound > > > autoLinks (inbound to the router from the broker) are established on > > > address myQueue, the broker(s) immediately provides initial credit on > > both > > > autoLinks. This initial credit is usually 500 or 1000 depending on the > > > broker you are using. > > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to > R2 > > > providing one credit each respectively to each router. > > > 3. Now the Producer connects to R1 and sends one message M1 (it cannot > > send > > > more than one message because the linkCapacity is set to 1). Assuming > M1 > > > goes to B1 (it has an equal chance of ending up in B2 as well), B1 > sees > > > that there are 2 consumers (autolinks) for that message.Say B1 sends M1 > > to > > > R1, M1 will be sent to C1. > > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or > R2. > > > 4a. If B2 sends M2 to R1, the message might go over the > > > inter-router link to R2 to C2 because there is no credit to send to > C1. > > > 4b. If B2 sends M2 to R2, then C2 gets M2 because routers > > prefer > > > local consumers to remote consumers. > > > 4c. If C1 is fast enough to consume M1 and replenish the > credit > > > and if B2 ends up sending M2 to R1, C1 might end up getting the second > > > message as well. > > > 4d. You need to also find out if the brokers load balances > > across > > > autolinks. > > > > > > In conclusion, you are lucky that the messages are being load balanced > > > across C1 and C2 when you changed linkCapacity to 1 but it is making it > > > more likely. This might not always happen. > > > > > > Turn on trace logging on the routers to see which route the messages > are > > > taking. Try the scenario over and over again. > > > > > > To turn on trace logging add the following to the router config file - > > > > > > log { > > > module: DEFAULT > > > enable: trace+ > > > output: path/to/qdrouterd.log > > > } > > > > > > If you want true load balancing between the consumers, put a router R3 > in > > > front of R1 and R2 and connect both consumers to R3. The producer can > > > connect to any router. > > > > > > > > > > > You can find attached the new config files. > > > > > > > > Is this the correct way to resolve this kind of problem? Does it > sound > > > > reasonable to you? > > > > > > > > Best regards, > > > > Rabih and Ali > > > > > > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[email protected]> > > > wrote: > > > > > > > >> > > > >> > > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[email protected]> > > > wrote: > > > >> > > > >>> Hello, > > > >>> > > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat > > > linux > > > >>> rhel 6.4. > > > >>> > > > >>> Use case description: > > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded > > queue > > > >>> "myQueue" on the 2 brokers. > > > >>> Here is an illustration: > > > >>> > > > >>> [image: Diagram.jpg] > > > >>> > > > >>> The producer produces 2 messages, the messages are load balanced on > > the > > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive(). > > > >>> > > > >> Does your receive() function issue one credit per call? > > > >> > > > >>> > > > >>> Our observation is that the consumer 1 consumes the first message > and > > > >>> the consumer 2 is never getting the second message. > > > >>> We are aware that the first dispatch router will do a prefetch for > > the > > > 2 > > > >>> messages but what is weird is that the prefetched message 2 is > never > > > routed > > > >>> to the second dispatch router and consumer 2. > > > >>> I attached the dispatch routers config. > > > >>> > > > >>> This is what is going on in my view - > > > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and > > > >> Message 2 goes to Broker 2. > > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing > one > > > >> credit. (The Consumer 2 has not yet attached to Router 2). The > > autolinks > > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a > > prefetch > > > of > > > >> 250 credits each. So,Message 1 and Message 2 > > > >> both come down to Router 1. Message 1 is sent to Consumer 1. > Message 2 > > > is > > > >> waiting on the outbound queue of Consumer 1 waiting to receive a > > credit > > > >> from Consumer 1. > > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that > > > >> message is already queued up to go to Consumer 1 who is still > keeping > > > his > > > >> connection open > > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back > to > > > >> its broker and that message will be sent to Consumer 2. > > > >> > > > >> Have you tried attaching Consumer 1 and Consumer 2 first and then > > > >> subsequently bringing up the Producer 1 and send two messages ? In > > that > > > >> case, I think, both consumers will receive one message. > > > >> > > > >> Thanks. > > > >> > > > >>> Do you have any idea how we can make consumer 2 receive the > message? > > > >>> > > > >>> Thanks for your help, > > > >>> Rabih > > > >>> > > > >>> > --------------------------------------------------------------------- > > > >>> To unsubscribe, e-mail: [email protected] > > > >>> For additional commands, e-mail: [email protected] > > > >> > > > >> > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
