So Colin was focusing on just the flow controller package. Most of the logic of moving messages 'offline' is in the activemq-queue module.
That package it self needs a good write up. But in general the goal is have queues which support the following options which can be enabled/disabled: * Paging: will the queue even attempt to spool to disk? In some high performance scenarios you may never want to spool messages to disk and instead prefer the producers to block when memory limits are reached. * Page Out Place Holders: If you don't page them out, then the queue will keep a list of pointers to the location of the message on disk in memory at all times. Keeping this list in memory will speed up accessing paged out messages as their location on disk is already known. Should be enabled for large queues so that even the message order and locations are determined by cursoring the persistence store. * Throttle Sources To Memory Limit : When disabled, the queue will behave very much like disabling flow control in 5.x. Fast producer's messages will be spooled to disk to avoid blocking the source. BTW the above is implemented in the CursoredQueue class if anyone is interested. We will need to review how best to implement that 'combined policy', but in the new architecture, I do think your going to see more 'temporary' producer blocking. For example, even with the 'Throttle Sources To Memory Limit' option disabled, the producers flow controller may block as the queue is trying to persist messages to the message store. This is because unlike 5.x, the message store is also flow controlled resource that is access async. On Thu, Jun 11, 2009 at 4:11 AM, Rob Davies<rajdav...@gmail.com> wrote: > Hi Colin, > > In 5.x flow control behaves as if its binary - off or on. When its off - > messages can be offlined (for non-persistent messages this means being > dumped to temporary storage) - but when its on - the producers slow and > stop. > Also - there can be cases when you get a temporary slow consumer (the > consuming app may be doing a big gc) - which means with flow control off - > messages get dumped to disk - and then the producers may never slow down > enough again for the consumer to catch up. Flow control is difficult to > implement for all cases - but we should allow for configuration of the > following: > > * maximum overall broker memory > * maximum memory allocation per destination > * maximum storage allocation > * maximum storage allocation per destination > * maximum temporary storage allocation > * maximum temporary storage allocation per destination > > when we start to hit a resource limit - we should aggressively gc messages > that have expired, then either offline (an flow control when that limit is > hit) or flow control. > It would be great to have a combined policy where we can block a producer > for a short time (seconds) then offline > For non-persistent messages - we still need a policy where we can remove > messages based on a selector (which would be in addition to expiring > messages). > > cheers, > > Rob >
