Hi Rob, Thanks for looking at this. We'll probably continue using single-queue consumers for now then, since the fair behavior is important to us, and try using multiple-queue consumers again when the new threading model is available.
Thanks, Helen On Tue, Jan 20, 2015 at 10:32 AM, Rob Godfrey <[email protected]> wrote: > Hi Helen, > > apologies for taking so long to respond > > On 6 January 2015 at 02:12, Helen Kwong <[email protected]> wrote: > > > Hi Rob, > > > > I finally got back to testing multiple queues on a consumer again, using > > the changes you added to help with fairness. My broker jvm is running > with > > -Dqueue.maxAsynchronousDeliveries=1 for the number of messages delivered > > per time slice by a queue. It has made things more fair than before, > though > > still significantly less so than before with single-queue consumers. Two > > fairness tests that we ran: > > > > A) In the test I described before, where a single session listens to 2 > > queues starting with 100 messages each, it is now indeed more fair. > > Sometimes there are 3 messages in a row from 1 queue, but the order is > > mostly alternating. When the first queue is drained, on average the > other / > > slower queue still has 6-7 messages left. However, this effect can > > accumulate and the number at the end can vary a lot -- I've seen up to > 26, > > which seems quite high compared to 100. Also ran this starting with 1000 > > messages per queue: on average the slower queue had about 22 messages > left > > at the end, and I've seen up to 65 messages left. > > > > B) We also ran tests with multiple connections / consumers listening to > the > > same set of queues. More specifically, we have 10 connections, each with > 1 > > consumer listening to 100 queues, each queue starting with 500 messages. > We > > look at the order in which messages are processed, and track what the > > largest difference in the number of remaining messages between any 2 > queues > > at any given point is. Before, in the analogous test where each > connection > > has 100 single-queue consumers for the 100 queues, the largest difference > > at any point is around 20. Now with multi-queue consumers, it's much > > higher, around 140-170. > > > > Our questions / concerns: > > > > 1. One thing I'm not sure about is whether it's possible that 1 queue can > > be favored over another queue. With this 1 consumer / 2 queues test, I > use > > the same 2 queues for every test run, one called BAREBONEQ0 and the other > > BAREBONEQ1. What I saw intially when running the test was that BAREBONEQ0 > > was almost always the slower queue, over maybe 20 or so test runs. But > > later I ran more test runs, and I don't see a pattern to who finishes > last > > anymore. It could be just by chance that I saw this in the beginning, but > > just to be sure I'm not missing anything, can one queue end up being > > favored over another at all? > > > > There shouldn't be any favouring - it will come down to thread scheduling > in the JVM at this point. The queues are "notified" of consumer credit in > a round robin order, with the "first" to be notified changed each time. > The JVM or operating system may simply favour a thread that is already > running. > > > > > > 2. With the second test, I'd often see a chunk of messages from the same > > queue given to different consumers around the same time. E.g., might see > 10 > > messages from one queue being processed consecutively, by the 10 > different > > consumers. 5 to 7 messages in a row from the same queue is common. Why > does > > this happen even though I have maxAsynchronousDeliveries configured to 1? > > You mentioned a queue pushes messages to a consumer when it's notified > that > > the consumer has room for messages -- what does this mean in terms of > > multiple consumers/connections, and does this mean whenever a queue gets > > notified, it may push 1 message to each consumer? > > > > > So, when I ran a similar test on my laptop I saw no more than about 3 or 4 > in a row from the same queue (before I changed how the round-robin > notification was done it was a lot worse)... again though this comes down > to thread scheduling... I was running everything (client / broker) on my 4 > core laptop... I imagine using a different machine would potentially see > different results. > > > > 3. Overall does the behavior from these tests seem expected to you? Is > this > > as fair as we can be with multi-queue consumers, before the new threading > > model allows consumers to pull from queues? > > > > > I made it as fair as I could with the current threading model... Keith and > I are reworking the IO / threading model at the moment though I still can't > really give an ETA for when that work will be finished - certainly not for > the release that will be cut shortly (15.02 or whatever we decide to call > it), possibly for the release after that though, > > Cheers, > Rob > > > > Thanks a lot! > > Helen > > > > On Fri, Oct 31, 2014 at 11:18 AM, Helen Kwong <[email protected]> > > wrote: > > > > > Thanks a lot for the changes and the explanation! Will try this > > workaround > > > for now. When the new queue threading model is available please let me > > know. > > > > > > On Fri, Oct 31, 2014 at 6:38 AM, Rob Godfrey <[email protected]> > > > wrote: > > > > > >> Hi Helen, > > >> > > >> so the fundamental issue here is that currently inside the broker > queues > > >> "push" messages to consumers rather than consumers pulling from > queues. > > >> When a queue is informed that a consumer has room to accept more > > messages, > > >> it immediately tries to start pump messages to that consumer. In this > > >> case > > >> there are two queues trying to pump messages to one consumer, and the > > >> first > > >> one to get notified will pump in up to 80 messages before yielding the > > >> thread. At this point the second queue might be able to jump in and > > pump > > >> 80 messages, or the first queue may actually get the lock again. > > >> > > >> Keith and I are planning on reworking the underlying queue threading > > model > > >> soon to change this around so that the consumers/connections pull from > > the > > >> queues, at which point real fairness will be easier to implement. In > > the > > >> meantime I've made a couple of changes (in QPID-6204, revision > > >> https://svn.apache.org/r1635768) which help by a) allowing the number > > of > > >> messages delivered in one "time-slice" to be configured on a per queue > > >> basis (i.e. removing the hardcoding of 80) and b) alternating which of > > the > > >> queues is notified first when the consumer has available credit. > > >> > > >> After this change, by setting the time slice to one delivery > > >> (-Dqueue.maxAsynchronousDeliveries=1) I saw reasonably fair behaviour > > >> (runs > > >> of no more the 3 messages for the same queue). Note that reducing the > > >> timeslice probably has some negative performance impact, but you can > > >> configure this value on a per queue basis (rather than setting it as a > > >> system property you can set on each individual queue as a context > > >> variable). > > >> > > >> When Keith gets done with what he's currently working on we'll try to > > >> update you on our work on changing the queue threading model around. > > >> > > >> Cheers, > > >> Rob > > >> > > >> > > >> On 31 October 2014 01:19, Helen Kwong <[email protected]> wrote: > > >> > > >> > Hi Rob, > > >> > > > >> > I got around to doing some testing on the multi-queue consumer > feature > > >> you > > >> > added. So far things have looked good mostly, but there is one issue > > >> I've > > >> > run into and would like your help on. > > >> > > > >> > When we had single-queue consumers, we had fair allocation behavior > > >> across > > >> > queues, in the sense that if I have 2 queues A and B, each with 100 > > >> > messages, and one JMS session having a listening consumer on queue A > > >> and a > > >> > listener on queue B, the message processing order will be round > robin > > -- > > >> > i.e., M_A_1 (representing the first message on queue A), M_B_1, > M_A_2, > > >> > M_B_2, M_A_3, M_B_3, and so on. But now, if I run the same test with > > the > > >> > session having a single multi-queue consumer on A and B instead, the > > >> order > > >> > is, roughly, first the 100 messages on A, followed by the messages > on > > B > > >> > (only a few B messages are processed before all A messages are > done). > > I > > >> > enqueue the messages in the round robin order. I've also tried this > > with > > >> > synchronous receives from both queues instead of asynchronous > > listening, > > >> > and I see similar behavior. > > >> > > > >> > Is there any way we can mimic the "fair" behavior of single-queue > > >> consumers > > >> > with multi-queue consumers? > > >> > > > >> > Thanks, > > >> > Helen > > >> > > > >> > > > > > > > > >
