Re: High load, lots of free memory and processes in devfs state
Does anybody have any ideas about this problem? So far today I've tried moving memcache off this machine, played with a few settings in httpd.conf but nothing seems to make a difference. Any help would be greatly appreciated. I'm rapidly losing what's left of my hair!! -Stut On 15 Jun 2008, at 18:18, Stut wrote: Hi, I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd 26780 80 1 -40 79892K 10036K devfs 0 0:00 1.51% httpd 26801 80 1 -40 81292K 11104K devfs 3 0:00 1.50% httpd 26786 80 1 -40 80456K 10796K devfs 1 0:00 1.41% httpd 26784 80 1 -40 79892K 10036K devfs 0 0:00 1.41% httpd 26785 80 1 -40 81304K 11228K devfs 0 0:00 1.41% httpd 26763 80 1 -40 83220K 13752K devfs 1 0:00 1.24% httpd ...etc... As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. When it's like this it's impossible to do anything - every command can take anything from a few seconds to a few minutes to respond - and the web user experience is shot to pieces. I can't find a reference that explains what the devfs state indicates. SA shows the following... [EMAIL PROTECTED]:~# sa 150107 242633.24re 1548.32cp 13avio 801k 31174 192785.02re 1488.13cp 32avio 502k httpd* 7047 6286.71re 21.53cp 67avio 2854k php 7703 164.83re 14.65cp3avio 743k convert 8 19.82re9.47cp 5016avio1k gzip 7703 177.66re7.70cp2avio 1352k composite 4 18.90re3.69cp1avio3k mysqldump 90.80re0.76cp 23avio 144k bzip2 64.16re0.53cp22998avio 1731k find 2890 1126.31re0.45cp0avio 175988k perl* 5415 40.83re0.32cp 11avio 123676k rateup 300914948.72re0.32cp3avio 9512k smtp 247 1812.85re0.27cp 440avio 945k cleanup 7 163.69re0.18cp0avio 105k top 167 1809.71re0.11cp 183avio 1117k pickup 2667 797.34re0.07cp0avio29671k trivial- rewrite 1820 96.40re0.02cp0avio 141901k ps 17 1482.98re0.02cp1avio 1083k sshd* 35629 6318.10re0.01cp0avio 18016816k sh 15232 391.68re0.01cp0avio 935430k sendmail 599 1223.30re0.01cp 28avio44065k bounce 70.95re0.01cp8avio 596k sa 35 1952.66re0.01cp 11avio16013k ***other 250.07re0.01cp 18avio 1447k rm 15232 203.94re0.00cp7avio 2994436k postdrop 611.20re0.00cp5avio34481k git 3 123.65re0.00cp 12avio 796k mail 92 345.77re0.00cp0avio34405k scache 312.13re0.00cp 66avio 7115k newsyslog 6 2140.32re0.00cp 14avio 6835k bash 5463 290.01re0.00cp0avio 2584397k grep 5 1479.06re0.00cp 17avio 3878k sshd 1830 99.75re0.00cp0avio 1221440k wc 42.86re0.00cp2avio13472k less 30.46re0.00cp0avio10048k pgrep 68 27.44re0.00cp0avio69632k sh* 35.06re0.00cp1avio
Re: High load, lots of free memory and processes in devfs state
Stut [EMAIL PROTECTED] wrote: I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd [...] Please let vmstat 5 run for a minute ... Anything that looks unusual? Have you checked dmesg? Is this FreeBSD i386 (32bit) or amd64 (64bit)? Have you considered updating? 6.2-RELEASE isn't the freshest anymore. You might even consider going to 7-stable and using the new ULE scheduler which copes better with SMP servers. As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. Well, load 10 isn't that much for a 4-way SMP system. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Perl will consistently give you what you want, unless what you want is consistency. -- Larry Wall ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: High load, lots of free memory and processes in devfs state
On 16 Jun 2008, at 18:48, Oliver Fromme wrote: Stut [EMAIL PROTECTED] wrote: I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd [...] Please let vmstat 5 run for a minute ... Anything that looks unusual? procs memory pagedisks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733 3025 4 17 79 1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970 6374 3 26 71 0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454 5131 3 26 71 1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076 4415 3 25 72 0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998 3770 3 24 73 1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497 4339 2 26 72 1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895 6950 10 30 60 1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405 2830 7 19 74 0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740 2733 4 23 73 11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231 2617 4 26 70 1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572 3373 4 25 70 2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243 2770 6 25 69 1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063 2484 6 25 69 0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281 3399 7 25 68 1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405 3605 6 25 70 44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773 4998 6 26 69 1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550 5466 5 26 69 3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558 6135 5 26 69 1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027 6477 4 27 69 2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376 6527 4 26 70 1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088 10242 6 28 66 1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407 8204 4 26 70 2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526 6728 3 26 70 1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081 6864 3 27 70 1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309 8996 5 27 68 Lots of processes blocked - which I guess is what the devfs state indicates. Have you checked dmesg? Yes. The only odd thing I can see is the following message, but from what I've read it's not critical until you get 5 and it's only in there once. collecting pv entries -- suggest increasing PMAP_SHPGPERPROC| Is this FreeBSD i386 (32bit) or amd64 (64bit)? FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/ sys/SMP amd64 Have you considered updating? 6.2-RELEASE isn't the freshest anymore. You might even consider going to 7-stable and using the new ULE scheduler which copes better with SMP servers. I have, but I'd rather understand what's happening. As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. Well, load 10 isn't that much for a 4-way SMP system. A couple of weeks ago this server was fairly fast, load never really going beyond 3 and everything was reasonably responsive. Now it regularly goes up to and beyond a load of 10 (more often than not at the moment) and everything is slow. This means our website users are getting a very poor experience which is reflected in our traffic levels which
Re: High load, lots of free memory and processes in devfs state
Is the MySQL daemon still running on that box? I see a mysqldump but no mysqld. If it is, try doing a shutdown and see if the load decreases. Sounds odd, but I have been having similar issues with MySQL. -Patrick 2008/6/16 Stut [EMAIL PROTECTED]: On 16 Jun 2008, at 18:48, Oliver Fromme wrote: Stut [EMAIL PROTECTED] wrote: I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd [...] Please let vmstat 5 run for a minute ... Anything that looks unusual? procs memory pagedisks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733 3025 4 17 79 1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970 6374 3 26 71 0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454 5131 3 26 71 1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076 4415 3 25 72 0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998 3770 3 24 73 1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497 4339 2 26 72 1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895 6950 10 30 60 1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405 2830 7 19 74 0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740 2733 4 23 73 11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231 2617 4 26 70 1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572 3373 4 25 70 2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243 2770 6 25 69 1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063 2484 6 25 69 0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281 3399 7 25 68 1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405 3605 6 25 70 44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773 4998 6 26 69 1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550 5466 5 26 69 3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558 6135 5 26 69 1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027 6477 4 27 69 2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376 6527 4 26 70 1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088 10242 6 28 66 1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407 8204 4 26 70 2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526 6728 3 26 70 1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081 6864 3 27 70 1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309 8996 5 27 68 Lots of processes blocked - which I guess is what the devfs state indicates. Have you checked dmesg? Yes. The only odd thing I can see is the following message, but from what I've read it's not critical until you get 5 and it's only in there once. collecting pv entries -- suggest increasing PMAP_SHPGPERPROC| Is this FreeBSD i386 (32bit) or amd64 (64bit)? FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SMP amd64 Have you considered updating? 6.2-RELEASE isn't the freshest anymore. You might even consider going to 7-stable and using the new ULE scheduler which copes better with SMP servers. I have, but I'd rather understand what's happening. As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. Well, load 10 isn't that much for a 4-way SMP system. A couple of weeks ago this server was fairly fast, load never really going beyond 3
Re: High load, lots of free memory and processes in devfs state
On 16 Jun 2008, at 22:20, Patrick C wrote: Is the MySQL daemon still running on that box? I see a mysqldump but no mysqld. If it is, try doing a shutdown and see if the load decreases. Sounds odd, but I have been having similar issues with MySQL. It's not, no. It used to but it doesn't run now. The mysqldump is a nightly backup. -Stut 2008/6/16 Stut [EMAIL PROTECTED]: On 16 Jun 2008, at 18:48, Oliver Fromme wrote: Stut [EMAIL PROTECTED] wrote: I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd [...] Please let vmstat 5 run for a minute ... Anything that looks unusual? procs memory pagedisks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 412 2 8910132 2743568 2230 3 1 0 2106 93 0 0 1098 4733 3025 4 17 79 1 412 0 8841792 2760548 1630 0 0 0 2276 0 21 0 658 3970 6374 3 26 71 0 422 0 8804744 2765076 1349 0 0 0 1394 0 12 0 576 3454 5131 3 26 71 1 421 0 8756808 2778660 1095 0 0 0 1581 0 48 0 574 3076 4415 3 25 72 0 420 0 8684932 2800128 2049 0 0 0 2927 0 45 0 505 2998 3770 3 24 73 1 380 0 8603676 2838452 1149 0 0 0 2842 0 22 0 505 3497 4339 2 26 72 1 21 0 8280260 3097392 7630 0 0 0 19969 0 29 0 1241 8895 6950 10 30 60 1 4 0 8195828 3131992 9113 0 0 0 9742 0 78 0 996 9405 2830 7 19 74 0 20 0 8169288 3121688 2899 0 0 0 2098 0 47 0 1193 7740 2733 4 23 73 11 13 0 8123476 3128688 1947 0 0 0 2160 0 52 0 1161 6231 2617 4 26 70 1 14 0 8067304 3143984 2298 0 0 0 2885 0 57 0 1806 5572 3373 4 25 70 2 17 1 8015924 3156168 2702 0 0 0 3164 0 23 0 1384 8243 2770 6 25 69 1 22 0 7956476 3176376 2013 0 0 0 2879 0 23 0 917 7063 2484 6 25 69 0 35 0 7944760 3150452 4274 0 0 0 2806 0 21 0 1591 8281 3399 7 25 68 1 67 3 7903160 3158776 2043 0 0 0 2442 0 20 0 1095 6405 3605 6 25 70 44 69 0 7872504 3147192 2569 0 0 0 1712 0 81 0 1137 5773 4998 6 26 69 1 146 0 7849632 3135388 2095 0 0 0 1274 0 19 0 869 5550 5466 5 26 69 3 195 2 7825932 3122116 2482 0 0 0 1586 0 15 0 863 5558 6135 5 26 69 1 244 3 7798148 3111624 1609 0 0 0 1226 0 10 0 849 4027 6477 4 27 69 2 273 3 7776772 3102948 2080 0 0 0 1310 0 14 0 604 4376 6527 4 26 70 1 312 2 7768232 3079012 3148 0 0 0 1800 0 12 0 1066 6088 10242 6 28 66 1 340 0 7742680 3068244 2020 0 0 0 1421 0 74 0 729 4407 8204 4 26 70 2 366 0 7740324 3068068 1612 0 0 0 1553 0 11 0 613 3526 6728 3 26 70 1 397 1 7928688 3059900 1886 0 0 0 1344 0 12 0 515 3081 6864 3 27 70 1 400 0 8074560 3008988 4771 0 0 0 1457 0 14 0 950 5309 8996 5 27 68 Lots of processes blocked - which I guess is what the devfs state indicates. Have you checked dmesg? Yes. The only odd thing I can see is the following message, but from what I've read it's not critical until you get 5 and it's only in there once. collecting pv entries -- suggest increasing PMAP_SHPGPERPROC| Is this FreeBSD i386 (32bit) or amd64 (64bit)? FreeBSD harold.freeads.co.uk 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/ SMP amd64 Have you considered updating? 6.2-RELEASE isn't the freshest anymore. You might even consider going to 7-stable and using the new ULE scheduler which copes better with SMP servers. I have, but I'd rather understand what's happening. As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. Well, load 10 isn't that
High load, lots of free memory and processes in devfs state
Hi, I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 26807 80 1 -40 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -40 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -40 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -40 82392K 12640K devfs 3 0:00 1.84% httpd 26780 80 1 -40 79892K 10036K devfs 0 0:00 1.51% httpd 26801 80 1 -40 81292K 11104K devfs 3 0:00 1.50% httpd 26786 80 1 -40 80456K 10796K devfs 1 0:00 1.41% httpd 26784 80 1 -40 79892K 10036K devfs 0 0:00 1.41% httpd 26785 80 1 -40 81304K 11228K devfs 0 0:00 1.41% httpd 26763 80 1 -40 83220K 13752K devfs 1 0:00 1.24% httpd ...etc... As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. When it's like this it's impossible to do anything - every command can take anything from a few seconds to a few minutes to respond - and the web user experience is shot to pieces. I can't find a reference that explains what the devfs state indicates. SA shows the following... [EMAIL PROTECTED]:~# sa 150107 242633.24re 1548.32cp 13avio 801k 31174 192785.02re 1488.13cp 32avio 502k httpd* 7047 6286.71re 21.53cp 67avio 2854k php 7703 164.83re 14.65cp3avio 743k convert 8 19.82re9.47cp 5016avio1k gzip 7703 177.66re7.70cp2avio 1352k composite 4 18.90re3.69cp1avio3k mysqldump 90.80re0.76cp 23avio 144k bzip2 64.16re0.53cp22998avio 1731k find 2890 1126.31re0.45cp0avio 175988k perl* 5415 40.83re0.32cp 11avio 123676k rateup 300914948.72re0.32cp3avio 9512k smtp 247 1812.85re0.27cp 440avio 945k cleanup 7 163.69re0.18cp0avio 105k top 167 1809.71re0.11cp 183avio 1117k pickup 2667 797.34re0.07cp0avio29671k trivial- rewrite 1820 96.40re0.02cp0avio 141901k ps 17 1482.98re0.02cp1avio 1083k sshd* 35629 6318.10re0.01cp0avio 18016816k sh 15232 391.68re0.01cp0avio 935430k sendmail 599 1223.30re0.01cp 28avio44065k bounce 70.95re0.01cp8avio 596k sa 35 1952.66re0.01cp 11avio16013k ***other 250.07re0.01cp 18avio 1447k rm 15232 203.94re0.00cp7avio 2994436k postdrop 611.20re0.00cp5avio34481k git 3 123.65re0.00cp 12avio 796k mail 92 345.77re0.00cp0avio34405k scache 312.13re0.00cp 66avio 7115k newsyslog 6 2140.32re0.00cp 14avio 6835k bash 5463 290.01re0.00cp0avio 2584397k grep 5 1479.06re0.00cp 17avio 3878k sshd 1830 99.75re0.00cp0avio 1221440k wc 42.86re0.00cp2avio13472k less 30.46re0.00cp0avio10048k pgrep 68 27.44re0.00cp0avio69632k sh* 35.06re0.00cp1avio13120k showq 59 109.55re0.00cp0avio 243008k smtpd 1830.54re0.00cp2avio 518400k unlink 21.61re0.00cp 33avio 4608k whereis 120.00re0.00cp0avio35000k adjkerntz 50
