https://issues.apache.org/bugzilla/show_bug.cgi?id=44846
Summary: High load accumulates CLOSE_WAITs until server is
unusable under Event MPM
Product: Apache httpd-2
Version: 2.2.8
Platform: PC
OS/Version: Linux
Status: NEW
Severity: major
Priority: P2
Component: Event MPM
AssignedTo: [email protected]
ReportedBy: [EMAIL PROTECTED]
While testing event MPM, using ab on another machine, if I pick
-c NNN (concurrency) greater than MaxClients, then although the
benchmark finishes reasonably, dangling CLOSE_WAIT connections
are left with Recv-Q data pointing to the httpd parent process.
Repeating the same highly concurrent ab a few more times causes
the CLOSE_WAIT connections to accumulate further until eventually
there are so many that httpd is no longer reachable at all.
When reaches this state, even after many minutes (or longer?)
it remains unreachable.. it must be killed (which ages the
CLOSE_WAITS very quickly) and restarted.
With lower concurrency the decay in performance to unreachability
is much longer but still present, and CLOSE_WAITs still
accumulate roughly in proportion to how slow the server gets.
<IfModule event.c>
ListenBackLog 32
StartServers 10
ThreadLimit 50
ThreadsPerChild 50
MinSpareThreads 100
MaxSpareThreads 1000
ThreadsPerChild 50
MaxClients 500
MaxRequestsPerChild 0
</IfModule>
KeepAlive is off.
Initial ab:
/ab -c 1000 -n 5000 http://192.168.1.1/server-status
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.1 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Finished 5000 requests
Server Software: Apache
Server Hostname: 192.168.1.1
Server Port: 80
Document Path: /server-status
Document Length: 215 bytes
Concurrency Level: 1000
Time taken for tests: 0.977488 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Non-2xx responses: 5014
Total transferred: 1900306 bytes
HTML transferred: 1078010 bytes
Requests per second: 5115.15 [#/sec] (mean)
Time per request: 195.498 [ms] (mean)
Time per request: 0.195 [ms] (mean, across all concurrent requests)
Transfer rate: 1897.72 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 15 10.7 13 41
Processing: 4 67 122.3 21 684
Waiting: 2 64 122.5 18 682
Total: 6 83 127.4 34 723
Percentage of the requests served within a certain time (ms)
50% 34
66% 45
75% 65
80% 83
90% 257
95% 279
98% 676
99% 680
100% 723 (longest request)
Subsequent abs get lower and lower RPS ..
./ab -c 1000 -n 10000 http://192.168.1.1/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
Server Software: Apache
Server Hostname: 192.168.1.1
Server Port: 80
Document Path: /front/xxx
Document Length: 0 bytes
Concurrency Level: 1000
Time taken for tests: 17.232351 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2370474 bytes
HTML transferred: 0 bytes
Requests per second: 580.30 [#/sec] (mean)
Time per request: 1723.235 [ms] (mean)
Time per request: 1.723 [ms] (mean, across all concurrent requests)
Transfer rate: 134.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 84 745.6 1 9023
Processing: 13 162 830.0 24 14539
Waiting: 11 160 830.0 22 14538
Total: 14 246 1150.7 25 16639
Percentage of the requests served within a certain time (ms)
50% 25
66% 30
75% 41
80% 52
90% 140
95% 669
98% 3271
99% 6669
100% 16639 (longest request)
Until telnet ab or lynx just hang on the port.
And from netstat:
netstat -nap|grep CLOSE_WAIT | grep 192.168.1.1:80| more
tcp 96 0 192.168.1.1:80 192.168.1.30:49689 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:57625 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:37144 CLOSE_WAIT
31414/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:57111 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:37143 CLOSE_WAIT
31414/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:57110 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:37142 CLOSE_WAIT
31414/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:51989 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:37141 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37141 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:51988 CLOSE_WAIT
-
tcp 96 0 192.168.1.1:80 192.168.1.30:37140 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37138 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37137 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37138 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37137 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37136 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37135 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37134 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37133 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37132 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:54283 CLOSE_WAIT
31469/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37131 CLOSE_WAIT
31532/httpd
tcp 96 0 192.168.1.1:80 192.168.1.30:37130 CLOSE_WAIT
31532/httpd
(etc)
Notice there are even some older entries from past runs where the server was
killed and restarted..
the kernel is 2.6.11 for this test, using epoll
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
