On Fri, Jul 12, 2019 at 11:29 AM Rabih M <[email protected]> wrote:
> Hello,
>
> Thank you Ganesh for your answer.
> To answer your first question yes the consumers we have, issue one credit
> per call.
> For the the second question, yes we are attaching consumers before sending
> the messages.
>
> After setting the dispatch "link capacity" on the listeners to 1, we
> managed to have an equal load-balancing of the messages between consumer 1
> and 2.
> Our understanding is that the problem comes from the prefetch at the level
> of the "normal" listener, which is preventing the "inter-router" listener
> to pass messages to the second dispatch.
>
Just want to note here that the linkCapacity on the inter-router listeners
are ignored. So setting linkCapacity to 1 on that listener has no effect.
This is what I think is happening
1. The Router 1 (R1) and Router 2 (R2) are started and each of them connect
to both Brokers, B1 and B2 and establish the autolinks. As soon as inbound
autoLinks (inbound to the router from the broker) are established on
address myQueue, the broker(s) immediately provides initial credit on both
autoLinks. This initial credit is usually 500 or 1000 depending on the
broker you are using.
2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
providing one credit each respectively to each router.
3. Now the Producer connects to R1 and sends one message M1 (it cannot send
more than one message because the linkCapacity is set to 1). Assuming M1
goes to B1 (it has an equal chance of ending up in B2 as well), B1 sees
that there are 2 consumers (autolinks) for that message.Say B1 sends M1 to
R1, M1 will be sent to C1.
4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
4a. If B2 sends M2 to R1, the message might go over the
inter-router link to R2 to C2 because there is no credit to send to C1.
4b. If B2 sends M2 to R2, then C2 gets M2 because routers prefer
local consumers to remote consumers.
4c. If C1 is fast enough to consume M1 and replenish the credit
and if B2 ends up sending M2 to R1, C1 might end up getting the second
message as well.
4d. You need to also find out if the brokers load balances across
autolinks.
In conclusion, you are lucky that the messages are being load balanced
across C1 and C2 when you changed linkCapacity to 1 but it is making it
more likely. This might not always happen.
Turn on trace logging on the routers to see which route the messages are
taking. Try the scenario over and over again.
To turn on trace logging add the following to the router config file -
log {
module: DEFAULT
enable: trace+
output: path/to/qdrouterd.log
}
If you want true load balancing between the consumers, put a router R3 in
front of R1 and R2 and connect both consumers to R3. The producer can
connect to any router.
>
> You can find attached the new config files.
>
> Is this the correct way to resolve this kind of problem? Does it sound
> reasonable to you?
>
> Best regards,
> Rabih and Ali
>
> On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <[email protected]> wrote:
>
>>
>>
>> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <[email protected]> wrote:
>>
>>> Hello,
>>>
>>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat linux
>>> rhel 6.4.
>>>
>>> Use case description:
>>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue
>>> "myQueue" on the 2 brokers.
>>> Here is an illustration:
>>>
>>> [image: Diagram.jpg]
>>>
>>> The producer produces 2 messages, the messages are load balanced on the
>>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
>>>
>> Does your receive() function issue one credit per call?
>>
>>>
>>> Our observation is that the consumer 1 consumes the first message and
>>> the consumer 2 is never getting the second message.
>>> We are aware that the first dispatch router will do a prefetch for the 2
>>> messages but what is weird is that the prefetched message 2 is never routed
>>> to the second dispatch router and consumer 2.
>>> I attached the dispatch routers config.
>>>
>>> This is what is going on in my view -
>> 1. The producer sends two messages - Message 1 goes to Broker 1 and
>> Message 2 goes to Broker 2.
>> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
>> credit. (The Consumer 2 has not yet attached to Router 2). The autolinks
>> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch of
>> 250 credits each. So,Message 1 and Message 2
>> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2 is
>> waiting on the outbound queue of Consumer 1 waiting to receive a credit
>> from Consumer 1.
>> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
>> message is already queued up to go to Consumer 1 who is still keeping his
>> connection open
>> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
>> its broker and that message will be sent to Consumer 2.
>>
>> Have you tried attaching Consumer 1 and Consumer 2 first and then
>> subsequently bringing up the Producer 1 and send two messages ? In that
>> case, I think, both consumers will receive one message.
>>
>> Thanks.
>>
>>> Do you have any idea how we can make consumer 2 receive the message?
>>>
>>> Thanks for your help,
>>> Rabih
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
