> Can you grab a stack trace from each of the httpd processes (e.g.,
> with pstack, or by scripting gdb) to see what they're doing when
> the load drops to zero?
Yes, but... pstack does not find any symbols (gdb does). Any way around
this? Am I compiling apache incorrectly?
Also, I like this idea. I'll write a script to run and wait until the
idle time is over 60% and then start logging a pstack of all apache
processes. I have no clue how to do that, but I'll figure it out.
> Also, does netstat show connections piling up in 'established'
> state on the port on which your httpd is listening?
Yes, even while idle is high. What does this mean???
Note that the load goes way down while the established connections rise
-- for several seconds. No processes are waiting for run time. Then,
bang! High load and hundreds of processes waiting for run time and then
lots waiting for run and lots in 'uninterruptable sleep'. It is madness!
Load goes from 5-15 to 60-220!
Remember, connections here will include both those used by Apache to the
client (no Keep-Alive) and by Apache to the MySQL server (persistent is
yes). Max clients for Apache is 410, so the max connections should be
820-860 or so for Apache. (There is a named running using a few as
well).
while (echo 1); do vmstat 1 2; netstat --statistics | grep 'connections
est';done
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
5 0 0 114300 97428 4672 49144 28 50 88 75 111 357 59
22 19
8 0 0 114300 97436 4684 49180 0 0 40 16 10173 8251 46
17 38
365 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
12 0 1 114300 97436 4684 49180 28 50 88 75 112 357 59
22 19
35 0 1 114300 97436 4704 49316 0 0 156 0 12269 8041 71
22 7
356 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
13 0 2 114300 97436 4712 49336 28 50 88 75 112 357 59
22 19
11 0 0 114300 97436 4712 49344 8 0 16 0 12383 9121 58
22 21
345 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
7 0 0 114300 97436 4716 49344 28 50 88 75 112 358 59
22 19
7 0 1 114300 97964 4784 49404 0 0 96 188 11222 9017 50
21 29
332 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
3 0 0 114300 98204 4788 49404 28 50 88 75 112 358 59
22 19
3 0 0 114300 99344 4792 49432 4 0 40 0 10784 9045 44
18 39
326 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
5 0 1 114300 99344 4792 49432 28 50 88 75 112 358 59
22 19
1 0 0 114300 99776 4800 49432 12 0 12 40 9543 7702 43
15 42
324 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
4 0 0 114300 99776 4800 49432 28 50 88 75 113 358 59
22 19
10 0 0 114300 100636 4816 49608 8 0 200 0 10150 8890 38
14 48
333 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
2 0 1 114300 100720 4816 49608 28 50 88 75 113 358 59
22 19
12 0 0 114300 101372 4820 49628 0 0 24 0 10896 8939 51
19 30
331 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
5 0 0 114300 101372 4820 49628 28 50 88 75 113 358 59
22 19
4 0 0 114300 101100 4868 50096 0 0 516 0 11971 9779 55
21 24
329 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
1 0 0 114300 101100 4868 50096 28 50 88 75 113 358 59
22 19
1 0 0 114296 101660 4908 50140 4 0 52 120 9255 7912 43
15 42
321 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114296 101760 4908 50140 28 50 88 75 113 358 59
22 19
2 0 0 114296 102880 4924 50144 8 0 16 52 11214 9160 53
17 30
323 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
2 0 0 114296 102912 4924 50144 28 50 88 75 113 359 59
22 19
5 0 0 114296 102920 4944 50232 0 0 112 0 12258 9168 60
23 17
329 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
3 0 0 114296 102920 4944 50232 28 50 88 75 114 359 59
22 19
2 0 0 114296 102792 4960 50372 0 0 156 0 11192 8830 48
20 32
337 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
4 0 0 114296 102792 4960 50372 28 50 88 75 114 359 59
22 19
4 0 0 114296 102792 4960 50404 0 0 32 0 12109 9525 54
21 25
326 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
1 0 0 114296 102792 4960 50404 28 50 88 75 114 359 59
22 19
7 0 0 114260 103908 4968 50472 8 0 40 80 11363 9657 49
18 33
328 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
7 0 0 114260 103908 4968 50472 28 50 88 75 114 359 59
22 19
0 0 0 114260 104204 4976 50544 28 0 108 0 10663 8804 52
18 30
325 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 104204 4976 50544 28 50 88 75 114 359 59
22 19
2 0 0 114260 104668 4988 50596 0 0 64 0 10596 8753 43
19 39
326 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
2 0 0 114260 104668 4988 50596 28 50 88 75 115 359 59
22 19
8 1 1 114260 105468 5016 50652 28 0 120 0 11163 9331 48
16 36
336 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
8 0 1 114260 105468 5016 50652 28 50 88 75 115 360 59
22 19
3 0 0 114260 105364 5032 50688 0 0 60 0 12541 10173 55
22 23
331 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
1 0 0 114260 105364 5032 50688 28 50 88 75 115 360 59
22 19
7 0 0 114260 105260 5048 50704 12 0 40 16 10463 8565 50
16 34
326 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
2 0 0 114260 105260 5048 50704 28 50 88 75 115 360 59
22 19
11 0 0 114260 106616 5064 50776 0 0 88 0 12155 8915 59
23 18
332 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
1 0 0 114260 106616 5064 50776 28 50 88 75 115 360 59
22 19
5 0 0 114260 106476 5080 50944 0 0 180 0 9364 7202 59
15 25
327 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
2 0 1 114260 106456 5080 50944 28 50 88 75 115 360 59
22 19
54 0 0 114260 106360 5084 50976 0 0 36 0 3534 2440 14
6 80
368 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
13 0 0 114260 106556 5084 51008 28 50 88 75 115 360 59
22 19
0 0 0 114260 106600 5092 51116 4 0 124 0 9835 6055 45
19 37
385 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 106600 5092 51116 28 50 88 75 116 360 59
22 19
13 0 0 114260 106848 5108 51280 0 0 172 36 5808 4296 28
10 62
424 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
6 0 1 114260 106848 5108 51280 28 50 88 75 116 360 59
22 19
0 0 0 114260 106768 5212 51280 44 0 44 512 4289 3384 23
9 68
475 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 106768 5212 51280 28 50 88 75 116 360 59
22 19
0 0 0 114260 106752 5212 51280 0 0 0 0 2087 1067 4
2 94
541 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 106752 5212 51280 28 50 88 75 116 360 59
22 19
0 0 0 114260 105380 5216 51312 0 0 36 0 1760 732 5
3 92
613 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 105276 5216 51312 28 50 88 75 116 360 59
22 19
0 0 0 114260 100388 5224 51312 0 0 0 16 2126 1326 7
4 89
625 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 100212 5224 51312 28 50 88 75 116 360 59
22 19
11 0 0 114260 89200 5228 51308 16 0 20 0 6728 5178 26
13 61
643 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 89040 5228 51308 28 50 88 75 116 360 59
22 19
0 0 0 114260 68260 5228 51316 0 0 8 0 7964 5749 35
19 46
684 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 68152 5228 51316 28 50 88 75 116 360 59
22 19
0 0 0 114260 49980 5276 51348 0 0 28 236 3434 1994 17
12 71
748 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
1 0 0 114260 49960 5244 51380 28 50 88 75 116 360 59
22 19
0 0 0 114260 32508 5268 51420 0 0 24 204 3050 1770 15
12 74
807 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 32284 5268 51420 28 50 88 75 116 360 59
22 19
0 1 0 114260 12964 5308 51436 0 0 24 152 3271 1958 16
12 73
854 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 12808 5308 51436 28 50 88 75 116 361 59
22 19
0 0 0 114260 11080 5296 51448 0 0 0 0 1460 611 3
2 94
853 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
0 0 0 114260 10976 5296 51448 28 50 88 75 116 361 59
22 19
146 0 0 114260 8572 5300 51448 0 0 4 0 1581 415 5
2 92
842 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
51 224 3 128012 6916 1956 22188 28 50 88 75 118 361 59
22 19
28 223 3 128012 8052 1988 20312 88 0 380 0 12617 7623
72 27 0
795 connections established
1
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
22 224 3 128012 7876 2012 20548 28 50 88 75 118 362 59
22 19
35 223 3 128012 7368 2036 20688 44 0 208 0 13472 7846
71 29 0
794 connections established
