On Wed, 2007-08-22 at 17:25 +0100, Andrew Stitcher wrote:
> On Wed, 2007-08-22 at 08:02 -0400, Alan Conway wrote:
> > You were wondering about what data structure was growing so big? Guess
> > what: it's the Serializer again. The problem is not the serializer
> > per-se, its a fairness problem.
>
> I think I understand a (little) more about the cause of this problem
> now. I only see the problem if I give the broker 1 netio thread. In this
> case the serializer thread can run on the other processor and in effect
> you get more resources devoted to I than O. when I run with 1 netio
> threads I don't see this problem as the I and O can run in parallel (on
> different sockets anyway) and the serializer competes with each of them
> equally. Well it's something along these lines I'm sure. Eliminating the
> extra thread of the serializer should help.
>
> Incidentally I don't see any significant speed up, but thats pr just the
> clients taking up the slack.
>
> I'm still seeing this with your latest changes BTW, so it's not wholly
> due to the allocations by the serializer.
>
> Andrew
The following are just observations that might be relevant to your
thinking, I'm not doing any more analysis till your new threading model
is in:
In my latest checkins I only removed allocations due to boost::functor
in the serializer. There's still the std::deque allocating memory. Lots
of memory, 258M for a 500000 perftest run to be precise. While the
functor was worse in terms of number of allocations, the deque is (much)
worse in terms of total allocated space:
258.4 60.5% 60.5% 258.4 60.5% __gnu_cxx::new_allocator::allocate
87.7 20.6% 81.1% 95.4 22.3%
qpid::broker::BrokerAdapter::BasicHandlerImpl::publish
63.9 15.0% 96.1% 63.9 15.0% std::basic_string::_Rep::_S_create
7.6 1.8% 97.9% 7.6 1.8% boost::detail::shared_count::shared_count
See attached diagram which shows the call chains.
Imagine for a moment that the broker is a black box and we don't know
how it works. If it doesn't lose messages we can say with certainty that
whenever messages arrive faster than they leave then *some* data
structure in there has to be holding the difference. So some of this
type of queueing is normal broker activity.
On the other hand if there are consumers available and capable of
pulling messages as fast as the producer writes them, we'd expect that
broker never holds very many in memory because it can pass them off
right away.
I think thats not happening now: the broker is stacking up a large
number of messages in a "read burst" which pile up in the deque before
it can write them out again. It looks like we need some form of flow
control within the broker.
Cheers,
Alan.
