Greg Ames wrote: >when I came in to work today, top on daedalus showed all three load >averages above 27. vmstat -w 5 showed that the number of processes in >the run queue was jumping up to over 350 fairly frequently, and there's >some evidence of spikes in CPU usage. So I bounced us back to httpd >2_0_28, and the bad behavior has definately gotten better. > >Either we're doing something to make more processes run-able at times, >or httpd related processes are staying in the run queue longer than they >used to sometimes. I think it's the latter. For one thing, the idle >CPU drops down to zero periodically, which I've never seen with 2_0_28. > >Jeff & I discussed how to troubleshoot this baby. Some thoughts: >* compare trusses and look for abnormalities >* scrutinize the error logs >* calculate and log how much CPU time each request takes, the thought >being that some requests are burning a lot more CPU than they used to. > >Any other ideas? >
Do you have a way to take a snapshot of each httpd process's stack backtrace? On Solaris, I'd do this by running /usr/proc/bin/pstack on each pid; I don't know if FreeBSD has a similar functionality. This would give us a picture of what all those runnable processes are doing. --Brian
