Optimising PrefetchSubscription.dispatchPending() ideas

[David Sitsky]

In my application, I have noticed with 20 consumers, the broker's CPU is 
going through the roof, with many threads in 
PrefetchSubscription.dispatchPending().  With my consumers, it might be 
500-1000 messages dispatched before a commit() can be called.  With 20 
consumers, this means there can be a build-up of 20,000 uncommited 
messages lying around the system, let-alone the new messages which are 
being pumped into the system at a furious rate.  Not nice I know, but I 
don't have much choice about it at the moment, for application-specific 
reasons.

As you can imagine, I can have some very big pending queue sizes - 
sometimes 100,000 in size.

I am experimenting with different prefetch sizes which may help, but I 
suspect every time a prefetch thread is trying to dispatch a message, it 
might have to iterate through very large numbers of deleted messages or 
messages which have been claimed by other subscribers before it finds a 
matching message.  Multiply this by 20, and there is a lot of CPU being 
consumed.  This worries me for scalability reasons - if I want to keep 
bumping up the number of consumers.

I'm not sure what the best way of improving this is... is it possible 
when we call dispatchPending() to not call pendingMessageCursor.reset() 
perhaps?

I'm trying to understand why we need to reset the cursor, when 
presumably all off the messages we have gone over before in a previous 
dispatchPending() call are either deleted, dispatched or locked by 
another node, and therefore don't need to be checked again (or we check 
if we reach the end of the cursor list)?

I realise if a transaction is rolled back, that a message that was 
previously locked by another consumer may be freed.  There are probably 
message ordering isues too.

Is it possible when we are iterating through the cursor if we find a 
node locked by another consumer to perhaps move it to the end of the 
cursor (or another list) and check it only if we found no matches?

I'm sure there are a lot of complexities here I am not aware of - but I 
am curious what others think.

Doing this sort of chance should reduce the latencies and CPU usage of 
the broker significantly.

Cheers,
David