On Thu, 2007-08-23 at 10:02 +0100, Gordon Sim wrote: > Alan Conway wrote: > > On the other hand if there are consumers available and capable of > > pulling messages as fast as the producer writes them, we'd expect that > > broker never holds very many in memory because it can pass them off > > right away. > > > > I think thats not happening now: the broker is stacking up a large > > number of messages in a "read burst" which pile up in the deque before > > it can write them out again. It looks like we need some form of flow > > control within the broker. > > Prior to the serializer, the publishing thread dispatched messages > directly where consumers were available. This had the effect of slowing > down the publication (by tying it to the consumption). > > The serializer as currently implemented has a dedicated thread per queue > that does the dispatch; the publisher simpling pushing its message onto > the deque.
Not so. The Serializer will continue dispatch in the calling thread *unless* doing so might cause tasks to be processed out of order - i.e. if there is already a thread doing a dispatch or there are already tasks queued up. (If it doesn't its a bug.) It was designed to allow reader threads to continue all the way thru when possible, and to return (rather than blocking on a mutex as before) when not possible. The intent was to avoid reader threads blocking, and it was too successful. As soon as a thread hits a busy serializer, it drops its load and returns. As long as the serializer is busy, the read thread are now doing nothing but decoding, which of course they can do faster than the broker can process and encode them. > If the serializer's thread can't push out the message as fast as they > come in (when consumers are available) then I think we should focus on > making that part faster/more efficient (as previously mentioned I think > having this work done on the consumer IO thread would allow us to remove > a lot of locking and could get us some gains here). What we need is a fair distribution of work and/or flow control to slow down faster elements while slower elements catch up. The Serializer has given reader threads an unfair advantage. > We do want flow control in the broker to allow the size of queues to be > managed (where the rate of consumption is slowed by the application > rather than the broker code), but in this particular case slowing down > the publisher because the process of dispatching to the consumers can't > keep up seems like a retrograde step. Agreed. What I think we need (and I think Andrew is working towards) is to eliminate the serializer and make the queues themselves the only serialization and handover point for thereads. Only at the queues do we have the tools to do useful flow control. Ideally we should allow a single thread to go all the way from read to write when possible. That would be self balancing. May not be easy to do though Cheers, Alan.
