On Mon, 19 Dec 2005, Perrin Harkins wrote:
processes free for accepting jobs (as opposed to processing jobs). If
there are not, it adds the job to the queue and goes back to listening
for requests. If there are, it processes the job. This ensures that
processing jobs does not starve the ability to accept new ones.
Be careful with this approach because this idea, while sweet and peachy on
paper, is often fundamentally wrong in practice. My place of employment
implemented complex call processing software based on queues percisely
because "processing jobs does not starve the ability to accept new ones".
I considered that the implementation would have terrible consequences for
performance and behavior under overload, and my coworkers strongly
disagreed. So I armed a test:
(call generator) -> (our software) -> (call receiver)
At 20 CPS, our software ate 20% of the CPU and 20% of the Memory. Thus it
wouldn't be all that unreasonable to expect 30 CPS to be possible. As it
turned out, problems of lock contention were harsh enough that attempting
the rate of call attempts per second on the call generator meant that the
call receiver would only get 5 call attempts per second.
But that failure was much more about lock contention than it was about
application workflow. The real demonstration was yet to come.
I fetched one of our developers back to our desk to point out the
contention issues. Rather than re-starting all the software, I told the
call generator to "pause" - ie, stop making new phone calls.
Stunningly, the moment I did this, the call receiver went from 5 call
attempts per second to 10, and kept receiving call attempts for several
minutes (even though they were all invalid and the generator had been
stopped long ago). What was happening? The application had been taking
messages into the queue, promising the call generator to handle them. Thus
the queue kept growing, and growing, and growing...
Now, a queue is of course applicable in some places. The application
mentioned in this thread is one of them. But IMNSHO, you should never ever
think that a queue's ability to allow you to accept new work while you're
busy processing current work is a good thing unless this trait is (a)
necessary for performance and (b) carefully constrained to keep things
under control (ie, the kernel's network buffers).
- Perrin
Cheers,
Chase
