Thanks for the explanation Robbie. On Wed, Jan 4, 2012 at 1:12 PM, Robbie Gemmell <[email protected]>wrote:
> Hi Praveen, > > I can only really guess to any design decision on that front as it > would have been before my time with the project, but I'd say its > likely just that way because theres never been a strong need / use > case that actually required doing anything else. For example, with > most of the users I liase with the data they are using has at least > some degree of time sensitivity to it and having anywhere near that > volume of persistent data in the broker would represent some sort of > ongoing period of catastrophic failure in their application. I can > only really think of one group who make it into multi-million message > backlogs at all, and that usually includes having knowingly published > things which noone will ever consume. > > For a FIFO queue you are correct it would 'just' need to load in more > as required. Things get trickier when dealing with some of the other > queue types however, such as as LVQ/conflation and the recently added > Sorted queue types. Making the broker able to hold partial segments of > the queue in memory is something we have discussed doing in the past > for other reasons, but message volume hasnt really been a significant > factor in those considerations until now. I will take note of it for > any future work we do in that area though. > > Robbie > > On 1 January 2012 17:46, Praveen M <[email protected]> wrote: > > Hi, > > > > I was digging in the code base and was trying to understand how the > broker > > is implemented. > > I see that for each message enqueued there are certain objects kept in > > memory one for each message. > > > > example: MessageTransferReference, SimpleQueueEntryImpl etc. > > > > I tried computing the memory footprint of each individual message and it > > amounts about 320 bytes/message. > > I see that because of the footprint of each message, if i'm limited to > 4GB > > of memory, then I am limited to only about 13 million messages in the > > system at one point. > > > > Since I'm using a persistent store I'd have expected to go over 13 > million > > messages and be limited by disk store rather than physical memory, but > > I realized this isn't the case. > > > > I am curious as to what were the driving points for this design decision > to > > keep a reference to every message in memory. I'd have expected in a FIFO > > queue you just need a subset of messages in memory and can pull in > > messages on demand rather than maintain reference to every message in > > memory. > > > > Can someone please explain as to the reasons for this design? Also, was > it > > assumed that we'd never flood the queues over 13 million messages at one > > time. Was there a bound > > decided upon? > > > > Thank you, > > Praveen > > > > > > > > -- > > -Praveen > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > > -- -Praveen
