On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[email protected]> wrote:
> Hello, > > We are testing with the trace+ log. We will brief you of the results when > they are ready. > > Our goal is not to load balance equally the messages between the consumers > but we would like the dispatch router to send the message to the free > consumer. > What you did (setting the linkCapacity on the producer and consumer listeners to 1) is the right thing to do if you want the router to send the message to *any* free consumer. If Consumer C1 is processing the first message and does not issue the next credit until it finishes processing the first message and if the second message arrives to Router 1 when C1 is still processing the message, then Router 1 will definitely forward the message to Router 2. BUT if C1 is fast enough and ends up processing the first message and immediately issues credit, the second message if it arrives in Router 1 will also be sent to C1 (because the router prefers local consumers). Remember that the key here is which Router ends up getting the message since each broker has two autoLinks to both routers and we don't know which autoLink the broker will choose. But overall, with your new configuration, the router will send the message to a consumer that is not busy no further configuration is necessary. Specially if the consumer does a long calculation, we do not want to block > the message in the outbound queue knowing there is another idle consumer. > > Is there any special configuration for this? > > Best regards, > Rabih > > > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[email protected]> wrote: > > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[email protected]> > wrote: > > > > > Hello, > > > > > > Thank you Ganesh for your answer. > > > To answer your first question yes the consumers we have, issue one > credit > > > per call. > > > For the the second question, yes we are attaching consumers before > > sending > > > the messages. > > > > > > After setting the dispatch "link capacity" on the listeners to 1, we > > > managed to have an equal load-balancing of the messages between > consumer > > 1 > > > and 2. > > > Our understanding is that the problem comes from the prefetch at the > > level > > > of the "normal" listener, which is preventing the "inter-router" > listener > > > to pass messages to the second dispatch. > > > > > Just want to note here that the linkCapacity on the inter-router > listeners > > are ignored. So setting linkCapacity to 1 on that listener has no effect. > > > > This is what I think is happening > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them > connect > > to both Brokers, B1 and B2 and establish the autolinks. As soon as > inbound > > autoLinks (inbound to the router from the broker) are established on > > address myQueue, the broker(s) immediately provides initial credit on > both > > autoLinks. This initial credit is usually 500 or 1000 depending on the > > broker you are using. > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2 > > providing one credit each respectively to each router. > > 3. Now the Producer connects to R1 and sends one message M1 (it cannot > send > > more than one message because the linkCapacity is set to 1). Assuming M1 > > goes to B1 (it has an equal chance of ending up in B2 as well), B1 sees > > that there are 2 consumers (autolinks) for that message.Say B1 sends M1 > to > > R1, M1 will be sent to C1. > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2. > > 4a. If B2 sends M2 to R1, the message might go over the > > inter-router link to R2 to C2 because there is no credit to send to C1. > > 4b. If B2 sends M2 to R2, then C2 gets M2 because routers > prefer > > local consumers to remote consumers. > > 4c. If C1 is fast enough to consume M1 and replenish the credit > > and if B2 ends up sending M2 to R1, C1 might end up getting the second > > message as well. > > 4d. You need to also find out if the brokers load balances > across > > autolinks. > > > > In conclusion, you are lucky that the messages are being load balanced > > across C1 and C2 when you changed linkCapacity to 1 but it is making it > > more likely. This might not always happen. > > > > Turn on trace logging on the routers to see which route the messages are > > taking. Try the scenario over and over again. > > > > To turn on trace logging add the following to the router config file - > > > > log { > > module: DEFAULT > > enable: trace+ > > output: path/to/qdrouterd.log > > } > > > > If you want true load balancing between the consumers, put a router R3 in > > front of R1 and R2 and connect both consumers to R3. The producer can > > connect to any router. > > > > > > > > You can find attached the new config files. > > > > > > Is this the correct way to resolve this kind of problem? Does it sound > > > reasonable to you? > > > > > > Best regards, > > > Rabih and Ali > > > > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[email protected]> > > wrote: > > > > > >> > > >> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[email protected]> > > wrote: > > >> > > >>> Hello, > > >>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat > > linux > > >>> rhel 6.4. > > >>> > > >>> Use case description: > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded > queue > > >>> "myQueue" on the 2 brokers. > > >>> Here is an illustration: > > >>> > > >>> [image: Diagram.jpg] > > >>> > > >>> The producer produces 2 messages, the messages are load balanced on > the > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive(). > > >>> > > >> Does your receive() function issue one credit per call? > > >> > > >>> > > >>> Our observation is that the consumer 1 consumes the first message and > > >>> the consumer 2 is never getting the second message. > > >>> We are aware that the first dispatch router will do a prefetch for > the > > 2 > > >>> messages but what is weird is that the prefetched message 2 is never > > routed > > >>> to the second dispatch router and consumer 2. > > >>> I attached the dispatch routers config. > > >>> > > >>> This is what is going on in my view - > > >> 1. The producer sends two messages - Message 1 goes to Broker 1 and > > >> Message 2 goes to Broker 2. > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one > > >> credit. (The Consumer 2 has not yet attached to Router 2). The > autolinks > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a > prefetch > > of > > >> 250 credits each. So,Message 1 and Message 2 > > >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2 > > is > > >> waiting on the outbound queue of Consumer 1 waiting to receive a > credit > > >> from Consumer 1. > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that > > >> message is already queued up to go to Consumer 1 who is still keeping > > his > > >> connection open > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to > > >> its broker and that message will be sent to Consumer 2. > > >> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and then > > >> subsequently bringing up the Producer 1 and send two messages ? In > that > > >> case, I think, both consumers will receive one message. > > >> > > >> Thanks. > > >> > > >>> Do you have any idea how we can make consumer 2 receive the message? > > >>> > > >>> Thanks for your help, > > >>> Rabih > > >>> > > >>> --------------------------------------------------------------------- > > >>> To unsubscribe, e-mail: [email protected] > > >>> For additional commands, e-mail: [email protected] > > >> > > >> > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > >
