On 07/23/2012 10:13 PM, Ted Ross wrote:
What we have is a store that is optimized for journaling (write-only) performance to support message persistence. Flow-to-disk is a completely different use case and should be implemented as a separate feature. The primary design goal should be to have a memory footprint that is not correlated to the queue size.
Agreed. It is more of a paging/swapping solution. You have a fixed amount of memory to use, with a file backing the queue providing extra space. You can then move the portion of that file that is mapped into memory at any time in order to deliver messages.
You can bring large chunks of the queue state into memory at a time (obviously this may displace other portions of the state) which should mean that normal consume operation remains efficient and fast.
You don't need to worry about the reliability of the swapped data - if messages need to be durably recorded that is still handled by the journal, orthogonal to the paging solution.
Obviously if you scroll through the entire queue, e.g. with a browser, then you would have to load each part in turn. Were you to do that with two browsers at different positions you would trigger a lot of thrashing.
The interaction with special queue types, LVQ and priority queue for example would likely need special code. In the first instance these would likely be mutually exclusive options.
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
