Does anybody have any ideas about this problem?

So far today I've tried moving memcache off this machine, played with a few settings in httpd.conf but nothing seems to make a difference.


Any help would be greatly appreciated. I'm rapidly losing what's left of my hair!!

-Stut

On 15 Jun 2008, at 18:18, Stut wrote:

Hi,

I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance.

Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server.

Top shows the following...

last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43
619 processes: 1 running, 618 sleeping
CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free
Swap: 8192M Total, 124K Used, 8192M Free

PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84% httpd 26780 80 1 -4 0 79892K 10036K devfs 0 0:00 1.51% httpd 26801 80 1 -4 0 81292K 11104K devfs 3 0:00 1.50% httpd 26786 80 1 -4 0 80456K 10796K devfs 1 0:00 1.41% httpd 26784 80 1 -4 0 79892K 10036K devfs 0 0:00 1.41% httpd 26785 80 1 -4 0 81304K 11228K devfs 0 0:00 1.41% httpd 26763 80 1 -4 0 83220K 13752K devfs 1 0:00 1.24% httpd
...etc...

As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. When it's like this it's impossible to do anything - every command can take anything from a few seconds to a few minutes to respond - and the web user experience is shot to pieces. I can't find a reference that explains what the devfs state indicates.

SA shows the following...

[EMAIL PROTECTED]:~# sa
 150107   242633.24re     1548.32cp       13avio      801k
  31174   192785.02re     1488.13cp       32avio      502k   httpd*
   7047     6286.71re       21.53cp       67avio     2854k   php
   7703      164.83re       14.65cp        3avio      743k   convert
      8       19.82re        9.47cp     5016avio        1k   gzip
   7703      177.66re        7.70cp        2avio     1352k   composite
      4       18.90re        3.69cp        1avio        3k   mysqldump
      9        0.80re        0.76cp       23avio      144k   bzip2
      6        4.16re        0.53cp    22998avio     1731k   find
   2890     1126.31re        0.45cp        0avio   175988k   perl*
   5415       40.83re        0.32cp       11avio   123676k   rateup
   3009    14948.72re        0.32cp        3avio     9512k   smtp
    247     1812.85re        0.27cp      440avio      945k   cleanup
      7      163.69re        0.18cp        0avio      105k   top
    167     1809.71re        0.11cp      183avio     1117k   pickup
2667 797.34re 0.07cp 0avio 29671k trivial- rewrite
   1820       96.40re        0.02cp        0avio   141901k   ps
     17     1482.98re        0.02cp        1avio     1083k   sshd*
  35629     6318.10re        0.01cp        0avio 18016816k   sh
  15232      391.68re        0.01cp        0avio   935430k   sendmail
    599     1223.30re        0.01cp       28avio    44065k   bounce
      7        0.95re        0.01cp        8avio      596k   sa
     35     1952.66re        0.01cp       11avio    16013k   ***other
     25        0.07re        0.01cp       18avio     1447k   rm
  15232      203.94re        0.00cp        7avio  2994436k   postdrop
     61        1.20re        0.00cp        5avio    34481k   git
      3      123.65re        0.00cp       12avio      796k   mail
     92      345.77re        0.00cp        0avio    34405k   scache
     31        2.13re        0.00cp       66avio     7115k   newsyslog
      6     2140.32re        0.00cp       14avio     6835k   bash
   5463      290.01re        0.00cp        0avio  2584397k   grep
      5     1479.06re        0.00cp       17avio     3878k   sshd
   1830       99.75re        0.00cp        0avio  1221440k   wc
      4        2.86re        0.00cp        2avio    13472k   less
      3        0.46re        0.00cp        0avio    10048k   pgrep
     68       27.44re        0.00cp        0avio    69632k   sh*
      3        5.06re        0.00cp        1avio    13120k   showq
     59      109.55re        0.00cp        0avio   243008k   smtpd
    183        0.54re        0.00cp        2avio   518400k   unlink
      2        1.61re        0.00cp       33avio     4608k   whereis
     12        0.00re        0.00cp        0avio    35000k   adjkerntz
     50      189.34re        0.00cp        0avio   165000k   anvil
    364        1.27re        0.00cp        0avio   947300k   atrun
      3        0.00re        0.00cp        0avio    53100k   awk
     10        0.05re        0.00cp        0avio   298800k   bash*
     27        4.15re        0.00cp        0avio    62000k   cat
      7        0.00re        0.00cp        0avio    44800k   cmp
     40        0.01re        0.00cp        2avio    79100k   cp
   2937     5227.93re        0.00cp        0avio  5143200k   cron*
      6        0.00re        0.00cp        0avio    22800k   date
    183       14.68re        0.00cp        2avio   578900k   dd
      5        0.00re        0.00cp        0avio    10600k   egrep
    132      241.91re        0.00cp        0avio   552400k   error
      6        0.05re        0.00cp        0avio        0k   expr
    245      444.67re        0.00cp        1avio  1181500k   flush
      3        0.00re        0.00cp        0avio        0k   hostname
      8        0.03re        0.00cp        0avio    21000k   id
      4        0.04re        0.00cp        0avio   173800k   ipcs
    183        0.49re        0.00cp        0avio   670000k   jot
     16        0.35re        0.00cp        1avio    65900k   ls
      3        1.52re        0.00cp       11avio    16300k   man
      2        0.06re        0.00cp        3avio     6800k   manpath
     10        0.00re        0.00cp        2avio    29200k   mktemp
      2        0.00re        0.00cp        0avio        0k   mount
   1287        4.64re        0.00cp        3avio  3717800k   mv
      4        0.12re        0.00cp        0avio    17600k   postqueue
     57       10.01re        0.00cp        0avio   242300k   proxymap
      7        4.19re        0.00cp        0avio    60500k   sed
      7        4.18re        0.00cp        0avio    10100k   sort
      7        0.05re        0.00cp        0avio     6100k   sysctl
      9        0.00re        0.00cp        0avio    15000k   tee
      4       26.15re        0.00cp        0avio     9300k   time
      2        0.48re        0.00cp        0avio    25400k   zcat

I'm fairly new to SA but I can't see anything in there that indicates a problem. If I'm missing something please let mw know.

If anyone has any ideas as to what might be causing this and how to fix it I'd appreciate it. I'm getting close to giving up and simply reinstalling it.

-Stut

--
http://stut.net/

_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Reply via email to