CPU is practically idle, not out of memory, local drive is idle.
I'd check the I/O on the nfs server, and even test catting files on the web servers that are hosted on the nfs server during the high load.
My guess from your data would be that either the nfs server is I/O bound, the network between them is having issues, or you don't have enough nfs processes/memory/etc on the nfs server to handle the number of requests.
I would start looking at this as an nfs issue rather than a webserver issue. -Steve adam fisher wrote:
Here is what I am seeing,sorry for the big print out. top top - 04:36:52 up 7 days, 1:05, 1 user, load average: 73.88, 62.88, 91.72 Tasks: 207 total, 1 running, 206 sleeping, 0 stopped, 0 zombie Cpu(s): 8.0%us, 0.8%sy, 0.0%ni, 84.4%id, 6.3%wa, 0.2%hi, 0.3%si, 0.0%st Mem: 2062096k total, 1834196k used, 227900k free, 137360k buffers Swap: 8193140k total, 88k used, 8193052k free, 1157808k cached 18833 apache 15 0 29240 11m 3084 S 4 0.6 0:18.77 httpd 18895 apache 15 0 29364 11m 3092 S 2 0.6 0:00.46 httpd 18831 apache 15 0 29364 11m 3216 S 2 0.6 0:02.86 httpd 18872 apache 15 0 29124 12m 4052 S 2 0.6 0:01.90 httpd 18867 apache 15 0 29316 12m 3872 D 1 0.6 0:01.51 httpd 18893 apache 15 0 28896 11m 3028 S 1 0.6 0:00.38 httpd 18884 apache 15 0 29332 11m 3088 S 1 0.6 0:00.90 httpd 18861 apache 15 0 29308 11m 3116 D 1 0.6 0:01.81 httpd 18888 apache 15 0 28904 11m 3084 S 1 0.6 0:00.59 httpd ps -aux | grep httpd apache 17255 0.0 0.5 29496 12072 ? D 04:27 0:00 /usr/sbin/httpd apache 17261 0.0 0.5 28840 11296 ? D 04:27 0:00 /usr/sbin/httpd apache 17267 0.2 0.5 29176 11860 ? D 04:27 0:01 /usr/sbin/httpd apache 17268 0.7 0.6 29552 12868 ? D 04:27 0:04 /usr/sbin/httpd apache 17456 1.1 0.6 29728 13168 ? S 04:27 0:06 /usr/sbin/httpd apache 17890 0.5 0.6 29928 12588 ? D 04:28 0:02 /usr/sbin/httpd apache 17893 0.0 0.5 29032 11548 ? D 04:28 0:00 /usr/sbin/httpd apache 17895 0.0 0.5 29184 11716 ? D 04:28 0:00 /usr/sbin/httpd apache 17896 0.0 0.5 28740 11256 ? D 04:28 0:00 /usr/sbin/httpd apache 17897 0.0 0.5 28912 11452 ? D 04:28 0:00 /usr/sbin/httpd apache 17904 0.3 0.5 29288 11876 ? D 04:28 0:01 /usr/sbin/httpd apache 17913 0.5 0.5 29316 11892 ? D 04:29 0:02 /usr/sbin/httpd apache 17923 0.1 0.5 29364 12052 ? D 04:29 0:00 /usr/sbin/httpd apache 17925 0.8 0.6 29336 12792 ? D 04:29 0:04 /usr/sbin/httpd apache 17935 0.2 0.6 29100 12664 ? D 04:29 0:01 /usr/sbin/httpd apache 17940 0.2 0.5 29328 11944 ? D 04:29 0:01 /usr/sbin/httpd apache 17954 0.2 0.5 29332 11904 ? D 04:30 0:00 /usr/sbin/httpd apache 17956 0.0 0.5 28400 10760 ? D 04:30 0:00 /usr/sbin/httpd apache 17957 0.3 0.5 29700 12240 ? D 04:30 0:01 /usr/sbin/httpd apache 17960 0.6 0.6 29416 12668 ? D 04:30 0:02 /usr/sbin/httpd apache 17961 0.0 0.5 28148 10608 ? D 04:30 0:00 /usr/sbin/httpd apache 17962 0.1 0.5 29120 11700 ? D 04:30 0:00 /usr/sbin/httpd apache 17964 0.0 0.5 28136 10424 ? D 04:30 0:00 /usr/sbin/httpd apache 17968 0.0 0.5 29184 11744 ? D 04:30 0:00 /usr/sbin/httpd apache 17970 1.8 0.6 29292 12704 ? S 04:30 0:07 /usr/sbin/httpd iostat -x 1 avg-cpu: %user %nice %system %iowait %steal %idle 2.00 0.00 0.00 0.00 0.00 98.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 11.00 0.00 6.00 0.00 136.00 22.67 0.00 0.50 0.17 0.10 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 17.00 0.00 136.00 8.00 0.01 0.29 0.06 0.10 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 5.97 0.00 1.49 0.00 0.00 92.54 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 8.00 0.00 2.00 0.00 80.00 40.00 0.00 0.50 0.50 0.10 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 10.00 0.00 80.00 8.00 0.01 0.90 0.10 0.10 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 88 189244 137948 1172700 0 0 0 4 30 30 1 0 99 0 0 The web root is located on an NFS share. I restarted NFS on this box just to make sure. When I restart httpd and the load average drops to around 10 or 11 I can browse the webpage just fine. It is when it gets to around 150 that I can't. Any help is appreciated Adam ----- Ryan Simpkins <[EMAIL PROTECTED]> wrote:On Wed, March 28, 2007 09:30, adam fisher wrote:I just brought up a new web server on fedora core. The load averappears to grow tothe size of the MaxClients which is 150. If I lower it then the loadaverage lowers.When I run ps -aux | grep httpd, I have almost every httpd processinuninterruptible sleep. Are these persistent connections that justaren't beingclosed or is it something else? Never seen this before, Adam/proc/loadavg The load average numbers give the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes. They are the same as the load average numbers given by uptime(1) and other programs. If top doesn't show high CPU utilization, run iostat -x 1 for a bit and see what your disk is doing. vmstat is also helpful here. -Ryan /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. *//* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
smime.p7s
Description: S/MIME Cryptographic Signature
/* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
