A few thoughts:
In analyzing a few spikes on our site in the last few days, a clear pattern
has emerged: the database spikes, and the database spikes induce a
corresponding spike on the mod_perl server about 2-6 minutes later(because
mod_perl requests start queuing up). This is exacerbated by the fact that as
the site slows down, folks start double and triple-clicking on links and
buttons, which of course just causes things to get much worse.
This has a few ramifications. If your pages are not homogeneous in database
usage (i.e., some pages are much heavier than others), then throttling by
number of connections or throttling based on webserver load doesn't help
that much. You need to throttle based on database server load. This requires
some sort of mechanism whereby the webserver can sample the load on the
database server and throttle accordingly. Currently, we just mount a common
NFS fileserver, sample every minute, and restart the webserver if db load is
too high, which works OK.
The best course of action, though, is to tune your database, homogenize your
pages, and buy a bigger box, which we're doing.
-Ed
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]On Behalf Of Perrin Harkins
Sent: Thursday, January 04, 2001 6:38 PM
To: Justin
Cc: [EMAIL PROTECTED]
Subject: Re: the edge of chaos
Justin wrote:
> Thanks for the links! But. I wasnt sure what in the first link
> was useful for this problem, and, the vacuum bots discussion
> is really a different topic.
> I'm not talking of vacuum bot load. This is real world load.
>
> Practical experiments (ok - the live site :) convinced me that
> the well recommended modperl setup of fe/be suffer from failure
> and much wasted page production when load rises just a little
> above *maximum sustainable throughput* ..
The fact that mod_proxy doesn't disconnect from the backend server when
the client goes away is definitely a problem. I remember some
discussion about this before but I don't think there was a solution for
it.
I think Vivek was correct in pointing out that your ultimate problem is
the fact that your system is not big enough for the load you're
getting. If you can't upgrade your system to safely handle the load,
one approach is to send some people away when the server gets too busy
and provide decent service to the ones you do allow through. You can
try lowering MaxClients on the proxy to help with this. Then any
requests going over that limit will get queued by the OS and you'll
never see them if the person on the other end gets tired of waiting and
cancels. It's tricky though, because you don't want a bunch of slow
clients to tie up all of your proxy processes.
It's easy to adapt the existing mod_perl throttling handlers to send a
short static "too busy" page when there are more than a certain number
of concurrent requests on the site. Better to do this on the proxy side
though, so maybe mod_throttle could do it for you.
- Perrin
