>Number: 1733
>Category: os-solaris
>Synopsis: performance on 50+ web servers drops off after 3-5 days.
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: apache
>State: open
>Class: sw-bug
>Submitter-Id: apache
>Arrival-Date: Mon Jan 26 12:40:00 PST 1998
>Last-Modified:
>Originator: [EMAIL PROTECTED]
>Organization:
apache
>Release: any
>Environment:
SunOS 5.5.1 Generic i86pc i386 i86pc runing apache 1.2
128mgs memory
pentium 200mx
>Description:
System runs for "X" days eventually eating up all memory ("X" depends on amount
of traffic)
Killing the process and restarting solves the problem for a few hours.
I have found this to be the same on all versions of apache.
memory does not seem to be returned to the os after httpd dies.
this causes performance to drom off over a few day peroid. eventually
the web server will become unstable and die.
My only course of action is to reboot. What do you suggest?
here are ps and vmstat outputs:
/usr/ucb/ps -aux
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
root 29511 4.2 0.7 992 828 pts/0 O 12:30:28 0:00 /usr/ucb/ps -aux
nobody 29409 0.6 1.1 2112 1328 ? S 12:29:36 0:00 /usr/local/etc/htt
nobody 29448 0.6 1.0 2060 1276 ? S 12:30:02 0:00 /usr/local/etc/htt
nobody 29411 0.6 1.0 2060 1276 ? S 12:29:37 0:00 /usr/local/etc/htt
nobody 29418 0.6 1.1 2100 1316 ? S 12:29:41 0:00 /usr/local/etc/htt
nobody 29412 0.5 1.1 2076 1292 ? S 12:29:37 0:00 /usr/local/etc/htt
nobody 29338 0.5 1.1 2084 1300 ? S 12:28:09 0:00 /usr/local/etc/htt
nobody 29443 0.5 1.1 2080 1296 ? S 12:30:00 0:00 /usr/local/etc/htt
nobody 29420 0.5 1.0 2072 1288 ? S 12:29:41 0:00 /usr/local/etc/htt
nobody 29419 0.5 1.0 2072 1288 ? S 12:29:41 0:00 /usr/local/etc/htt
nobody 29406 0.5 1.1 2080 1296 ? S 12:29:31 0:00 /usr/local/etc/htt
nobody 29463 0.5 1.1 2088 1304 ? S 12:30:11 0:00 /usr/local/etc/htt
nobody 29431 0.5 1.0 2068 1284 ? S 12:29:59 0:00 /usr/local/etc/htt
nobody 29466 0.5 1.0 2064 1280 ? S 12:30:12 0:00 /usr/local/etc/htt
nobody 29452 0.5 1.1 2152 1368 ? S 12:30:05 0:00 /usr/local/etc/htt
root 3 0.4 0.0 0 0 ? S Jan 22 35:25 fsflush
nobody 29447 0.4 1.0 2068 1284 ? S 12:30:02 0:00 /usr/local/etc/htt
nobody 29461 0.4 1.0 2068 1264 ? S 12:30:10 0:00 /usr/local/etc/htt
nobody 29390 0.4 1.1 2092 1308 ? S 12:29:06 0:00 /usr/local/etc/htt
nobody 29467 0.4 1.0 2060 1280 ? S 12:30:12 0:00 /usr/local/etc/htt
nobody 29474 0.4 1.0 2068 1284 ? S 12:30:14 0:00 /usr/local/etc/htt
nobody 29449 0.4 1.1 2152 1368 ? S 12:30:02 0:00 /usr/local/etc/htt
root 27423 0.4 1.1 1948 1392 ? S 11:55:01 0:06 /usr/local/etc/htt
nobody 29415 0.4 1.0 2060 1276 ? S 12:29:39 0:00 /usr/local/etc/htt
nobody 29464 0.4 1.0 2060 1272 ? S 12:30:11 0:00 /usr/local/etc/htt
nobody 29444 0.4 1.0 2068 1284 ? S 12:30:01 0:00 /usr/local/etc/htt
nobody 29468 0.4 1.1 2084 1300 ? S 12:30:12 0:00 /usr/local/etc/htt
nobody 29404 0.4 1.0 2068 1284 ? S 12:29:31 0:00 /usr/local/etc/htt
nobody 29413 0.4 1.1 2152 1340 ? S 12:29:37 0:00 /usr/local/etc/htt
nobody 29477 0.4 1.0 2068 1260 ? S 12:30:15 0:00 /usr/local/etc/htt
nobody 29405 0.4 1.0 2060 1276 ? S 12:29:31 0:00 /usr/local/etc/htt
nobody 29476 0.4 1.0 2072 1264 ? S 12:30:15 0:00 /usr/local/etc/htt
nobody 29407 0.4 1.1 2152 1368 ? S 12:29:31 0:00 /usr/local/etc/htt
nobody 29479 0.4 1.0 2060 1276 ? S 12:30:17 0:00 /usr/local/etc/htt
nobody 29430 0.3 1.0 2060 1276 ? S 12:29:49 0:00 /usr/local/etc/htt
nobody 29410 0.3 1.1 2076 1292 ? S 12:29:36 0:00 /usr/local/etc/htt
nobody 29496 0.3 1.0 2068 1284 ? S 12:30:22 0:00 /usr/local/etc/htt
nobody 29402 0.3 1.0 2072 1288 ? S 12:29:30 0:00 /usr/local/etc/htt
nobody 29422 0.3 1.0 2068 1284 ? S 12:29:43 0:00 /usr/local/etc/htt
nobody 29478 0.3 1.0 2080 1272 ? S 12:30:17 0:00 /usr/local/etc/htt
nobody 29491 0.3 1.0 2076 1288 ? S 12:30:21 0:00 /usr/local/etc/htt
nobody 29499 0.3 1.1 2080 1292 ? S 12:30:23 0:00 /usr/local/etc/htt
nobody 29493 0.3 1.0 2072 1280 ? S 12:30:22 0:00 /usr/local/etc/htt
nobody 29469 0.3 1.0 2060 1276 ? S 12:30:12 0:00 /usr/local/etc/htt
nobody 29391 0.3 1.0 2060 1276 ? S 12:29:07 0:00 /usr/local/etc/htt
nobody 29481 0.3 1.0 2060 1272 ? S 12:30:17 0:00 /usr/local/etc/htt
nobody 29408 0.2 1.1 2076 1292 ? S 12:29:34 0:00 /usr/local/etc/htt
nobody 29446 0.2 1.0 2092 1280 ? S 12:30:02 0:00 /usr/local/etc/htt
nobody 29424 0.2 1.0 2060 1276 ? S 12:29:43 0:00 /usr/local/etc/htt
nobody 29401 0.2 1.4 3408 1756 ? S 12:29:29 0:00 /usr/local/etc/htt
nobody 29400 0.1 1.1 2136 1352 ? S 12:29:18 0:00 /usr/local/etc/htt
nobody 29500 0.1 0.7 1964 804 ? S 12:30:23 0:00 /usr/local/etc/htt
root 178 0.1 0.7 1656 892 ? S Jan 22 0:02 /usr/sbin/nscd
nobody 29376 0.1 1.1 2080 1296 ? S 12:28:47 0:00 /usr/local/etc/htt
nobody 29416 0.1 1.1 2080 1296 ? S 12:29:40 0:00 /usr/local/etc/htt
nobody 29428 0.1 1.1 2164 1352 ? S 12:29:43 0:00 /usr/local/etc/htt
nobody 29334 0.1 1.1 3408 1408 ? S 12:27:57 0:00 /usr/local/etc/htt
nobody 29421 0.1 1.1 2128 1340 ? S 12:29:42 0:00 /usr/local/etc/htt
root 172 0.1 0.6 1408 724 ? S Jan 22 0:16 /usr/sbin/cron
nobody 29423 0.1 1.1 2164 1368 ? S 12:29:43 0:00 /usr/local/etc/htt
root 26827 0.1 0.6 924 752 pts/0 S 11:47:15 0:00 -csh
root 0 0.0 0.0 0 0 ? T Jan 22 0:00 sched
root 1 0.0 0.1 348 100 ? S Jan 22 0:08 /etc/init -
root 2 0.0 0.0 0 0 ? S Jan 22 0:00 pageout
root 113 0.0 0.4 1616 512 ? S Jan 22 0:00 /usr/sbin/rpcbind
root 115 0.0 0.0 1524 0 ? S Jan 22 0:00 /usr/sbin/keyserv
root 121 0.0 0.4 1680 448 ? S Jan 22 0:00 /usr/sbin/kerbd
root 130 0.0 0.4 1452 436 ? S Jan 22 0:00 /usr/sbin/in.named
root 134 0.0 0.7 1704 840 ? S Jan 22 0:03 /usr/sbin/inetd -s
root 137 0.0 0.4 1608 488 ? S Jan 22 0:00 /usr/lib/nfs/statd
root 139 0.0 0.4 1516 484 ? S Jan 22 0:00 /usr/lib/nfs/lockd
root 158 0.0 0.6 1784 736 ? S Jan 22 0:00 /usr/lib/autofs/au
root 162 0.0 0.6 1372 752 ? S Jan 22 0:00 /usr/sbin/syslogd
root 188 0.0 0.4 2472 444 ? S Jan 22 0:00 /usr/lib/lpsched
root 198 0.0 0.5 1344 552 ? S Jan 22 0:00 lpNet
root 200 0.0 0.3 748 396 ? S Jan 22 0:00 /usr/lib/utmpd
root 202 0.0 0.7 1952 852 ? S Jan 22 0:00 /usr/sbin/vold
root 240 0.0 0.5 1516 608 ? S Jan 22 0:00 /usr/lib/nfs/nfsd
root 242 0.0 0.5 1664 588 ? S Jan 22 0:00 /usr/lib/nfs/mount
root 251 0.0 0.5 1296 652 ? S Jan 22 0:00 /usr/lib/saf/sac -
root 252 0.0 0.5 1352 604 console S Jan 22 0:00 /usr/lib/saf/ttymo
root 254 0.0 0.6 1368 668 ? S Jan 22 0:00 /usr/lib/saf/ttymo
root 26825 0.0 0.8 1352 936 ? S 11:47:14 0:00 in.telnetd
nobody 29247 0.0 1.0 2068 1284 ? S 12:26:02 0:00 /usr/local/etc/htt
nobody 29337 0.0 1.0 2068 1284 ? S 12:28:04 0:00 /usr/local/etc/htt
nobody 29351 0.0 1.1 2160 1376 ? S 12:28:16 0:00 /usr/local/etc/htt
VMSTAT
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr f0 s0 s1 -- in sy cs us sy id
0 0 0 43536 15488 0 118 2 0 3 0 0 0 0 3 0 408 649 525 10 26 63
0 0 0 177920 6748 0 426 0 0 0 0 0 0 0 0 0 499 945 767 2 47 51
3 0 0 175208 5232 0 1086 0 0 0 0 0 0 0 0 0 504 1431 901 14 72 14
1 0 0 175120 5084 0 214 0 0 0 0 0 0 0 0 0 526 926 744 12 34 54
4 0 0 168056 2752 0 1206 0 0 0 0 0 0 0 1 0 561 1489 999 5 83 12
VMSTAT after the webserver all processes are killed.
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr f0 s0 s1 -- in sy cs us sy id
0 0 0 43576 15484 0 118 2 0 3 0 0 0 0 3 0 408 649 525 10 26 63
0 0 0 208088 23828 0 7 0 0 0 0 0 0 0 1 0 275 13 339 0 4 96
0 0 0 208088 23828 0 0 0 0 0 0 0 0 0 0 0 244 10 345 0 6 94
0 0 0 208088 23828 0 0 0 0 0 0 0 0 0 0 0 245 10 322 0 6 94
0 0 0 208088 23828 0 0 0 0 0 0 0 0 0 0 0 225 10 284 0 8 92
0 0 0 208088 23828 0 0 0 0 0 0 0 0 0 0 0 243 10 293 0 6 94
after a reboot (and webserver is running)
VMSTAT
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr f0 s0 s1 -- in sy cs us sy id
0 0 0 183196 72892 0 28 64 0 0 0 0 0 0 11 0 573 930 727 10 37 53
0 0 0 183192 72808 0 16 72 0 0 0 0 0 1 8 0 536 814 647 2 35 63
4 0 0 183728 73076 0 25 32 0 0 0 0 0 0 5 0 475 1047 697 8 39 53
0 0 0 183724 73012 0 13 72 0 0 0 0 0 0 8 0 464 606 495 6 22 72
0 0 0 184316 73316 0 24 0 0 0 0 0 0 0 1 0 384 714 574 6 22 72
PS -AUX
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
root 336 2.5 0.9 1288 1104 pts/1 O 12:36:45 0:00 /usr/ucb/ps -aux
nobody 299 0.6 1.0 2080 1284 ? S 12:35:47 0:00 /usr/local/etc/htt
nobody 274 0.5 1.0 2072 1276 ? S 12:35:43 0:00 /usr/local/etc/htt
nobody 305 0.5 1.1 2088 1292 ? S 12:35:48 0:00 /usr/local/etc/htt
nobody 283 0.5 1.1 2100 1304 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 270 0.5 1.0 2076 1284 ? S 12:35:42 0:00 /usr/local/etc/htt
nobody 281 0.5 1.0 2068 1272 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 296 0.5 1.1 2092 1296 ? S 12:35:46 0:00 /usr/local/etc/htt
nobody 280 0.5 1.0 2068 1272 ? S 12:35:44 0:00 /usr/local/etc/htt
nobody 298 0.5 1.1 2104 1308 ? S 12:35:47 0:00 /usr/local/etc/htt
nobody 282 0.5 1.0 2060 1268 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 301 0.5 1.0 2084 1288 ? S 12:35:47 0:00 /usr/local/etc/htt
nobody 229 0.5 1.1 2092 1300 ? S 12:35:37 0:00 /usr/local/etc/htt
nobody 302 0.5 1.0 2076 1280 ? S 12:35:47 0:00 /usr/local/etc/htt
nobody 300 0.5 1.1 2092 1296 ? S 12:35:47 0:00 /usr/local/etc/htt
nobody 279 0.4 1.0 2064 1268 ? S 12:35:44 0:00 /usr/local/etc/htt
nobody 284 0.4 1.0 2084 1288 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 304 0.4 1.1 2092 1296 ? S 12:35:48 0:00 /usr/local/etc/htt
nobody 292 0.4 1.0 2076 1280 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 337 0.4 1.0 2072 1244 ? S 12:36:45 0:00 /usr/local/etc/htt
nobody 289 0.3 1.0 2068 1272 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 287 0.3 1.1 2152 1356 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 335 0.3 1.0 2068 1268 ? S 12:36:39 0:00 /usr/local/etc/htt
nobody 272 0.3 1.1 2132 1336 ? S 12:35:42 0:00 /usr/local/etc/htt
nobody 275 0.3 1.0 2080 1284 ? S 12:35:43 0:00 /usr/local/etc/htt
nobody 223 0.3 1.0 2072 1272 ? S 12:35:37 0:00 /usr/local/etc/htt
nobody 306 0.3 1.0 2068 1272 ? S 12:35:48 0:00 /usr/local/etc/htt
nobody 291 0.3 1.0 2068 1272 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 294 0.3 1.0 2060 1264 ? S 12:35:46 0:00 /usr/local/etc/htt
nobody 231 0.3 1.0 2060 1268 ? S 12:35:38 0:00 /usr/local/etc/htt
root 3 0.2 0.0 0 0 ? S 12:34:45 0:00 fsflush
nobody 285 0.2 1.1 2136 1336 ? S 12:35:45 0:00 /usr/local/etc/htt
nobody 276 0.2 1.0 2068 1272 ? S 12:35:43 0:00 /usr/local/etc/htt
nobody 268 0.2 1.1 2160 1364 ? S 12:35:41 0:00 /usr/local/etc/htt
root 329 0.2 0.4 708 528 pts/0 S 12:35:58 0:00 script -a /out
nobody 269 0.2 1.0 2072 1276 ? S 12:35:42 0:00 /usr/local/etc/htt
nobody 290 0.2 1.1 2192 1396 ? S 12:35:45 0:00 /usr/local/etc/htt
root 218 0.1 1.1 1948 1344 ? S 12:35:37 0:00 /usr/local/etc/htt
root 330 0.1 0.4 732 456 pts/0 S 12:35:58 0:00 script -a /out
root 331 0.1 0.6 924 708 pts/1 S 12:35:58 0:00 sh -i
root 172 0.1 0.6 924 720 pts/0 S 12:35:28 0:00 -csh
root 205 0.1 1.1 1952 1380 ? S 12:35:32 0:00 /usr/sbin/vold
root 252 0.1 0.8 1664 980 ? S 12:35:39 0:00 /usr/lib/nfs/mount
root 134 0.0 0.9 1704 1104 ? S 12:35:16 0:00 /usr/sbin/inetd -s
root 250 0.0 0.7 1516 860 ? S 12:35:38 0:00 /usr/lib/nfs/nfsd
root 264 0.0 0.7 1368 904 ? S 12:35:40 0:00 /usr/lib/saf/ttymo
root 139 0.0 0.6 1516 740 ? S 12:35:17 0:00 /usr/lib/nfs/lockd
root 162 0.0 0.7 1372 836 ? S 12:35:18 0:00 /usr/sbin/syslogd
nobody 230 0.0 1.1 2124 1324 ? S 12:35:38 0:00 /usr/local/etc/htt
root 261 0.0 0.7 1296 832 ? S 12:35:39 0:00 /usr/lib/saf/sac -
root 262 0.0 0.7 1352 864 console S 12:35:39 0:00 /usr/lib/saf/ttymo
root 170 0.0 0.8 1388 976 ? S 12:35:26 0:00 in.telnetd
root 202 0.0 0.7 1344 872 ? S 12:35:32 0:00 lpNet
root 1 0.0 0.2 348 180 ? S 12:34:45 0:00 /etc/init -
root 113 0.0 0.5 1616 608 ? S 12:35:09 0:00 /usr/sbin/rpcbind
root 137 0.0 0.7 1608 848 ? S 12:35:16 0:00 /usr/lib/nfs/statd
root 158 0.0 0.9 1784 1140 ? S 12:35:18 0:00 /usr/lib/autofs/au
root 191 0.0 0.6 2472 780 ? S 12:35:31 0:00 /usr/lib/lpsched
root 0 0.0 0.0 0 0 ? T 12:34:45 0:00 sched
root 2 0.0 0.0 0 0 ? S 12:34:45 0:00 pageout
root 115 0.0 0.1 1524 24 ? S 12:35:09 0:00 /usr/sbin/keyserv
root 121 0.0 0.7 1680 864 ? S 12:35:10 0:00 /usr/sbin/kerbd
root 130 0.0 0.6 1420 656 ? S 12:35:16 0:00 /usr/sbin/in.named
root 175 0.0 0.6 1320 724 ? S 12:35:29 0:00 /usr/sbin/cron
root 181 0.0 0.8 1608 972 ? S 12:35:30 0:00 /usr/sbin/nscd
root 203 0.0 0.4 748 420 ? S 12:35:32 0:00 /usr/lib/utmpd
nobody 273 0.0 0.0 0 0 Z 0:00 <defunct>
nobody 277 0.0 0.0 0 0 Z 0:00 <defunct>
nobody 303 0.0 0.0 0 0 Z 0:00 <defunct>
>How-To-Repeat:
>Fix:
>Audit-Trail:
>Unformatted:
[In order for any reply to be added to the PR database, ]
[you need to include <[EMAIL PROTECTED]> in the Cc line ]
[and leave the subject line UNCHANGED. This is not done]
[automatically because of the potential for mail loops. ]
