Hello Ted, Like already discussed, with linkCapacity = 1 in the normal listener, messages are routed to idle consumers if any, or equally load balanced between busy consumers. This is already solving a big part of our problem.
However with this configuration, we are limiting all our cluster and in particular producer features. We might need to use asynchronous send in other use cases which will not be possible with linkCapacity equals to one. Having different linkCapacity for incoming and outgoing links might solve the problem. Do you think it's a good practice to do that ? And do you think it is feasible to have that without having different entrypoints (listener) for each dispatch router? Best Regards, Ali -----Original Message----- From: Rabih M <[email protected]> Sent: lundi 22 juillet 2019 17:15 To: [email protected] Subject: Re: Uneven distribution of messages Hello Ted, We cannot use the link-routes because we need to have multiple consumers connected to the same queue. Having multiple queues depending on the task execution time can be a good solution. We are thinking about another solution too: To have consumer that receives all the messages and dispatches them to the free workers and at the same time it can monitor the load to upscale and downscale the workers depending on the work received... Thanks for your help, Best regards, Rabih On Fri, Jul 19, 2019 at 6:02 PM Ted Ross <[email protected]> wrote: > On Fri, Jul 19, 2019 at 7:50 AM Rabih M <[email protected]> wrote: > > > Hello, > > > > Yes this is one of our production use cases, where we have low > > throughput but the processing of the messages can take from seconds > > to minutes (financial calculations). This is why it was bothering us > > to have an idle consumer while another is overloaded. > > > > You should consider using link-routes for your consumers. Link-routes > trade off the load balancing for tight control of credit end-to-end. > If a consumer is link-routed to a broker, the broker won't deliver a > message on that link until the consumer issues a credit, and then it > will only delivery as many messages as credits were issued. > > > > > > I understand that for the performance, we need to prefetch messages > > from broker to make them available in the dispatch, but what i did > > not understand is why inside a router we need to assign the > > prefetched > messages > > to a consumer connection and not waiting until the connected > > consumer issues a credit, knowing that from a performance point of > > view, the > costly > > action of "prefetching the messages through IO calls" was made. > > Is it because the complexity of the routing algorithm and the > > communications between the dispatch routers will increase? > > > > > When messages are sent into the router network, they are immediately > routed to a destination. The routers don't hold messages for later routing. > Also, the synchronization of real-time credit state across a network > for all addresses is not practical or scalable. > > > > Last point, we did the following test: > > A dispatch router have a connector with a LinkCapacity=250 connected > > to a broker and a listener with a LinkCapacity=1 connected to a consumer. > > > > The important link capacity in this scenario is the 250 as it controls > the router pre-fetch. The consumer's link capacity of 1 is not > relevant to this case. > > > > > > [image: Untitled Diagram.jpg] > > > > I can't see the diagram, but I think I get the idea. > > > > > > > > 1- the router 1 prefetches 250 message from the broker > > 2- the consumer issues a credit > > 3- the consumer receives a message from the router but does not > acknowledge > > 4- the consumer issues another credit > > 5- the consumer receives a message from the router but does not > > acknowledge again Steps 4 and 5 can be repeated until the 250 msgs > > are all transferred to the consumer > > > > This is consistent with there being only one consumer for the address on > the network at the time the broker sent the 250 messages. > > > > > > Is this an expected behavior? consumer 1 should not acknowledge before he > > can receive another message knowing that the link capacity of the > listener > > is 1? > > > > Best practice for acknowledgement is for the consumer to acknowledge > (settle) immediately after finishing the processing of the message (i.e. > once that message is no longer consuming memory or compute resources on the > host). This causes the settlement state of deliveries to be directly > related to consumer resources. Again, the link capacity of 1 is not having > any effect on the behavior of this scenario. > > > > > > Thanks for your explanations and help, > > > > Am I to understand that your case is this? You have a distributed work > queue in which the time-to-process is highly variable. Some messages are > processed quickly and others take much longer. You don't want to incur the > longer latency on messages that can be handled quickly if there are many > more fast messages than slow messages. > > Is it possible to know beforehand which messages are going to take long? > Could you put these on a different queue with a different address? > > > > Best regards, > > Rabih > > > > > > On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <[email protected]> wrote: > > > >> > >> > >> On Wed, Jul 17, 2019 at 12:00 PM Rabih M <[email protected]> > wrote: > >> > >>> Hello, > >>> > >>> We tested with LinkCapacity equal to 1 on the "normal" listener with > >>> debug level trace+, here are our findings: > >>> Our Cluster: > >>> [image: Diagram.jpg] > >>> We are using the broker-j, the consumer are connected to the dispatch > >>> routers before we start sending. > >>> > >>> For use case 1: > >>> 1- the producer sends a message. > >>> 2- Consumer 1 issues one credit, receives the message without > >>> acknowledging. > >>> 3- the producer sends another message. > >>> 4- Consumer 2 in auto-ack mode issues one credit and receives the > >>> message. > >>> 5- we repeated steps 3 and 4 ten times. > >>> 6- Consumer 1 acknowledges. > >>> The results were correct: all the messages were correctly distributed > to > >>> the idle consumer. > >>> > >>> For use case 2: > >>> 1- the producer send 10 messages while no credits were issued yet by > the > >>> consumers. > >>> 2- Consumer 1 issues one credit, receives a message without > >>> acknowledging. > >>> 3- Consumer 2 in auto-ack mode issues one credit and times out after 5 > >>> seconds if nothing is received. > >>> 4- we repeated step 3 eight times. > >>> 5- Consumer 1 acknowledges. > >>> The results were not as expected: 4 messages were blocked in the > >>> outbound queue of the consumer 1 and consumer 2 was able to receive > only 5 > >>> messages. > >>> We analysed the traces to follow the messages. We found that 4 messages > >>> were blocked in the dispatch 1. > >>> Conclusion: if no consumers are issuing credits (are busy) then the > >>> incoming messages will be pre-assigned automatically by the dispatch > router > >>> to the listeners (in a round robin way?). > >>> > >>> Is it an expected behavior in the dispatch router? is it not supposed > to > >>> wait for a credit to be issued before binding the message to an > outbound > >>> queue? > >>> > >> > >> Yes, this is the expected behavior. The router does not propagate each > >> individual credit from receiver to sender. It would be impractical to > do > >> so, would not scale well, and probably still wouldn't provide the > behavior > >> you expect. What the router does is to use delivery settlement as the > way > >> to control credit flow to producers. If the link capacity is 250, each > >> producer will be limited to 250 unsettled deliveries at any time. As > the > >> deliveries are settled, more credit is issued. This scales well in > large > >> networks, keeps a limit on the memory consumed by deliveries, and allows > >> for high delivery rates. > >> > >> Is this a real use case or are you experimenting to learn how the router > >> works? > >> > >> Under steady state flow, the messages will be delivered to the consumers > >> in proportion to the rate at which the consumers acknowledge (settle) > >> deliveries. If a consumer attaches a receiving link but withholds > credit, > >> the router network will route deliveries to that consumer in > anticipation > >> of credit being issued. It is an anti-pattern to attach a receiving > link > >> and stop issuing credit. Credit should be used to control the rate of > >> delivery. If you want to stop delivery, detach the receiver. > >> > >> If you really want a completely synchronous transfer across your > network, > >> you can set the link capacity on the broker connections to 1. This will > >> limit the number of in-flight unsettled deliveries on each incoming > >> auto-link to 1. It will be slow and will be prone to stalling, > especially > >> if your consumers withhold acknowledgement. > >> > >> If you want to check the paths of the messages, I attached the routers > >>> logs of the use case 2. > >>> > >>> Best regards, > >>> Rabih > >>> > >>> On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <[email protected]> > >>> wrote: > >>> > >>>> On Mon, Jul 15, 2019 at 1:26 PM Rabih M <[email protected]> > >>>> wrote: > >>>> > >>>> > Hello, > >>>> > > >>>> > We are testing with the trace+ log. We will brief you of the results > >>>> when > >>>> > they are ready. > >>>> > > >>>> > Our goal is not to load balance equally the messages between the > >>>> consumers > >>>> > but we would like the dispatch router to send the message to the > free > >>>> > consumer. > >>>> > > >>>> > >>>> What you did (setting the linkCapacity on the producer and consumer > >>>> listeners to 1) is the right thing to do if you want the router > >>>> to send the message to *any* free consumer. > >>>> > >>>> If Consumer C1 is processing the first message and does not issue the > >>>> next > >>>> credit until it finishes processing the first message and if the > second > >>>> message arrives to Router 1 when C1 is > >>>> still processing the message, then Router 1 will definitely forward > the > >>>> message to Router 2. BUT if C1 is fast enough and ends up processing > the > >>>> first message and immediately issues credit, the second message if it > >>>> arrives in Router 1 will also > >>>> be sent to C1 (because the router prefers local consumers). > >>>> > >>>> Remember that the key here is which Router ends up getting the message > >>>> since each broker has two autoLinks to both routers and we don't know > >>>> which > >>>> autoLink the broker will choose. > >>>> > >>>> But overall, with your new configuration, the router will send the > >>>> message > >>>> to a consumer that is not busy no further configuration is necessary. > >>>> > >>>> Specially if the consumer does a long calculation, we do not want to > >>>> block > >>>> > the message in the outbound queue knowing there is another idle > >>>> consumer. > >>>> > > >>>> > Is there any special configuration for this? > >>>> > > >>>> > Best regards, > >>>> > Rabih > >>>> > > >>>> > > >>>> > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <[email protected]> > >>>> wrote: > >>>> > > >>>> > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[email protected] > > > >>>> > wrote: > >>>> > > > >>>> > > > Hello, > >>>> > > > > >>>> > > > Thank you Ganesh for your answer. > >>>> > > > To answer your first question yes the consumers we have, issue > one > >>>> > credit > >>>> > > > per call. > >>>> > > > For the the second question, yes we are attaching consumers > before > >>>> > > sending > >>>> > > > the messages. > >>>> > > > > >>>> > > > After setting the dispatch "link capacity" on the listeners to > 1, > >>>> we > >>>> > > > managed to have an equal load-balancing of the messages between > >>>> > consumer > >>>> > > 1 > >>>> > > > and 2. > >>>> > > > Our understanding is that the problem comes from the prefetch at > >>>> the > >>>> > > level > >>>> > > > of the "normal" listener, which is preventing the "inter-router" > >>>> > listener > >>>> > > > to pass messages to the second dispatch. > >>>> > > > > >>>> > > Just want to note here that the linkCapacity on the inter-router > >>>> > listeners > >>>> > > are ignored. So setting linkCapacity to 1 on that listener has no > >>>> effect. > >>>> > > > >>>> > > This is what I think is happening > >>>> > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of > them > >>>> > connect > >>>> > > to both Brokers, B1 and B2 and establish the autolinks. As soon as > >>>> > inbound > >>>> > > autoLinks (inbound to the router from the broker) are established > on > >>>> > > address myQueue, the broker(s) immediately provides initial credit > >>>> on > >>>> > both > >>>> > > autoLinks. This initial credit is usually 500 or 1000 depending on > >>>> the > >>>> > > broker you are using. > >>>> > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects > >>>> to R2 > >>>> > > providing one credit each respectively to each router. > >>>> > > 3. Now the Producer connects to R1 and sends one message M1 (it > >>>> cannot > >>>> > send > >>>> > > more than one message because the linkCapacity is set to 1). > >>>> Assuming M1 > >>>> > > goes to B1 (it has an equal chance of ending up in B2 as well), B1 > >>>> sees > >>>> > > that there are 2 consumers (autolinks) for that message.Say B1 > >>>> sends M1 > >>>> > to > >>>> > > R1, M1 will be sent to C1. > >>>> > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1 > >>>> or R2. > >>>> > > 4a. If B2 sends M2 to R1, the message might go over the > >>>> > > inter-router link to R2 to C2 because there is no credit to send > >>>> to C1. > >>>> > > 4b. If B2 sends M2 to R2, then C2 gets M2 because > routers > >>>> > prefer > >>>> > > local consumers to remote consumers. > >>>> > > 4c. If C1 is fast enough to consume M1 and replenish the > >>>> credit > >>>> > > and if B2 ends up sending M2 to R1, C1 might end up getting the > >>>> second > >>>> > > message as well. > >>>> > > 4d. You need to also find out if the brokers load > balances > >>>> > across > >>>> > > autolinks. > >>>> > > > >>>> > > In conclusion, you are lucky that the messages are being load > >>>> balanced > >>>> > > across C1 and C2 when you changed linkCapacity to 1 but it is > >>>> making it > >>>> > > more likely. This might not always happen. > >>>> > > > >>>> > > Turn on trace logging on the routers to see which route the > >>>> messages are > >>>> > > taking. Try the scenario over and over again. > >>>> > > > >>>> > > To turn on trace logging add the following to the router config > >>>> file - > >>>> > > > >>>> > > log { > >>>> > > module: DEFAULT > >>>> > > enable: trace+ > >>>> > > output: path/to/qdrouterd.log > >>>> > > } > >>>> > > > >>>> > > If you want true load balancing between the consumers, put a > router > >>>> R3 in > >>>> > > front of R1 and R2 and connect both consumers to R3. The producer > >>>> can > >>>> > > connect to any router. > >>>> > > > >>>> > > > > >>>> > > > You can find attached the new config files. > >>>> > > > > >>>> > > > Is this the correct way to resolve this kind of problem? Does it > >>>> sound > >>>> > > > reasonable to you? > >>>> > > > > >>>> > > > Best regards, > >>>> > > > Rabih and Ali > >>>> > > > > >>>> > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy < > [email protected] > >>>> > > >>>> > > wrote: > >>>> > > > > >>>> > > >> > >>>> > > >> > >>>> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M < > >>>> [email protected]> > >>>> > > wrote: > >>>> > > >> > >>>> > > >>> Hello, > >>>> > > >>> > >>>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on > >>>> redhat > >>>> > > linux > >>>> > > >>> rhel 6.4. > >>>> > > >>> > >>>> > > >>> Use case description: > >>>> > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A > sharded > >>>> > queue > >>>> > > >>> "myQueue" on the 2 brokers. > >>>> > > >>> Here is an illustration: > >>>> > > >>> > >>>> > > >>> [image: Diagram.jpg] > >>>> > > >>> > >>>> > > >>> The producer produces 2 messages, the messages are load > >>>> balanced on > >>>> > the > >>>> > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a > receive(). > >>>> > > >>> > >>>> > > >> Does your receive() function issue one credit per call? > >>>> > > >> > >>>> > > >>> > >>>> > > >>> Our observation is that the consumer 1 consumes the first > >>>> message and > >>>> > > >>> the consumer 2 is never getting the second message. > >>>> > > >>> We are aware that the first dispatch router will do a prefetch > >>>> for > >>>> > the > >>>> > > 2 > >>>> > > >>> messages but what is weird is that the prefetched message 2 is > >>>> never > >>>> > > routed > >>>> > > >>> to the second dispatch router and consumer 2. > >>>> > > >>> I attached the dispatch routers config. > >>>> > > >>> > >>>> > > >>> This is what is going on in my view - > >>>> > > >> 1. The producer sends two messages - Message 1 goes to Broker 1 > >>>> and > >>>> > > >> Message 2 goes to Broker 2. > >>>> > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive() > >>>> issuing one > >>>> > > >> credit. (The Consumer 2 has not yet attached to Router 2). The > >>>> > autolinks > >>>> > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a > >>>> > prefetch > >>>> > > of > >>>> > > >> 250 credits each. So,Message 1 and Message 2 > >>>> > > >> both come down to Router 1. Message 1 is sent to Consumer 1. > >>>> Message 2 > >>>> > > is > >>>> > > >> waiting on the outbound queue of Consumer 1 waiting to receive > a > >>>> > credit > >>>> > > >> from Consumer 1. > >>>> > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but > >>>> that > >>>> > > >> message is already queued up to go to Consumer 1 who is still > >>>> keeping > >>>> > > his > >>>> > > >> connection open > >>>> > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED > >>>> back to > >>>> > > >> its broker and that message will be sent to Consumer 2. > >>>> > > >> > >>>> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and > then > >>>> > > >> subsequently bringing up the Producer 1 and send two messages ? > >>>> In > >>>> > that > >>>> > > >> case, I think, both consumers will receive one message. > >>>> > > >> > >>>> > > >> Thanks. > >>>> > > >> > >>>> > > >>> Do you have any idea how we can make consumer 2 receive the > >>>> message? > >>>> > > >>> > >>>> > > >>> Thanks for your help, > >>>> > > >>> Rabih > >>>> > > >>> > >>>> > > >>> > >>>> --------------------------------------------------------------------- > >>>> > > >>> To unsubscribe, e-mail: [email protected] > >>>> > > >>> For additional commands, e-mail: [email protected] > >>>> > > >> > >>>> > > >> > >>>> > > > > >>>> --------------------------------------------------------------------- > >>>> > > > To unsubscribe, e-mail: [email protected] > >>>> > > > For additional commands, e-mail: [email protected] > >>>> > > > >>>> > > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >> > >> > ******************************* This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
