Issue #11257 has been reported by Steven Seed.
----------------------------------------
Bug #11257: Puppetmaster accumulates CLOSE_WAIT connections in apache server
https://projects.puppetlabs.com/issues/11257
Author: Steven Seed
Status: Unreviewed
Priority: Normal
Assignee:
Category: passenger
Target version:
Affected Puppet version: 2.7.5
Keywords: close_wait, passenger, apache
Branch:
I have two nearly identically configured puppetmasters situated behind a load
balancer. Both are running puppet-server-2.7.5-1 on RHEL 6.1. Periodically one
of the two starts to accumulate CLOSE_WAIT connections on port 8140 through the
apache service until the maxclients count in apache is reached (in my case it
is set to 300). Once this happens, my server will no long accept connections.
The client connection have been closed, but the server keeps them open in the
CLOSE_WAIT state for some reason. Nothing is logged on either the client or
server side after this point. So far this has happened twice over the last
couple of months requiring me to restart the apache service.
Here are the relevant apache settings:
<IfModule worker.c>
StartServers 4
MaxClients 300
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
The puppetmaster is run through Passenger. Here are the relevant sections of my
rack.conf file:
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-3.0.9/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9
PassengerRuby /usr/bin/ruby
# tune passenger performance
PassengerHighPerformance on
PassengerMaxPoolSize 24
PassengerPoolIdleTime 300
PassengerMaxRequests 10000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off
Last line in the http error_log file prior to hitting the max connections...
[Wed Dec 07 08:22:34 2011] [error] server reached MaxClients setting, consider
raising the MaxClients setting
I attached a debugger to one of the httpd processes in the CLOSE_WAIT state.
Here are is the back trace:
(gdb) bt
#0 0x00007fdd35ea84c0 in __read_nocancel () at
../sysdeps/unix/syscall-template.S:82
#1 0x00007fdd299a8a8f in oxt::syscalls::read (fd=19, buf=0x7fff7d2a228e,
count=2) at ext/oxt/system_calls.cpp:111
#2 0x00007fdd2999ac40 in Passenger::readExact (fd=19, buf=0x7fff7d2a228e,
size=2, timeout=0x0) at ext/common/Utils/IOUtils.cpp:613
#3 0x00007fdd2995eb0d in Passenger::readUint16 (fd=19, output=@0x7fff7d2a233e,
timeout=0x0) at ext/common/Utils/MessageIO.h:124
#4 0x00007fdd29982078 in
Passenger::readArrayMessage<std::vector<std::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::allocator<std::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > > (
fd=19, output=std::vector of length 0, capacity 0, timeout=0x0) at
ext/common/Utils/MessageIO.h:239
#5 0x00007fdd2995ebf1 in Passenger::readArrayMessage (fd=19, timeout=0x0) at
ext/common/Utils/MessageIO.h:282
#6 0x00007fdd2995f236 in Passenger::MessageChannel::read (this=0x7fdd38c48310,
args=std::vector of length 0, capacity 0) at ext/common/MessageChannel.h:315
#7 0x00007fdd29973d93 in Passenger::ApplicationPool::Client::sendGetCommand
(this=0x7fdd386b3210, options=..., reply=std::vector of length 0, capacity 0)
at ext/common/ApplicationPool/Client.h:367
#8 0x00007fdd29976185 in Passenger::ApplicationPool::Client::get
(this=0x7fdd386b3210, options=...) at ext/common/ApplicationPool/Client.h:708
#9 0x00007fdd299791b5 in Hooks::getSession (this=0x7fdd38a1ab00, options=...)
at ext/apache2/Hooks.cpp:300
#10 0x00007fdd2997ad2e in Hooks::handleRequest (this=0x7fdd38a1ab00,
r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:670
#11 0x00007fdd29981318 in Hooks::handleRequestWhenInHighPerformanceMode
(this=0x7fdd38a1ab00, r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:1624
#12 0x00007fdd2995bbd7 in handle_request_when_in_high_performance_mode
(r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:1786
#13 0x00007fdd375f5980 in ap_run_handler (r=0x7fdd38c59038) at
/usr/src/debug/httpd-2.2.15/server/config.c:158
#14 0x00007fdd375f923e in ap_invoke_handler (r=0x7fdd38c59038) at
/usr/src/debug/httpd-2.2.15/server/config.c:376
#15 0x00007fdd37604880 in ap_process_request (r=0x7fdd38c59038) at
/usr/src/debug/httpd-2.2.15/modules/http/http_request.c:282
#16 0x00007fdd37601748 in ap_process_http_connection (c=0x7fdd38a26038) at
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
#17 0x00007fdd375fd488 in ap_run_process_connection (c=0x7fdd38a26038) at
/usr/src/debug/httpd-2.2.15/server/connection.c:43
#18 0x00007fdd37609657 in child_main (child_num_arg=<value optimized out>) at
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:667
#19 0x00007fdd3760996a in make_child (s=0x7fdd38523860, slot=254) at
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:763
#20 0x00007fdd3760a5ec in perform_idle_server_maintenance (_pconf=<value
optimized out>, plog=<value optimized out>, s=<value optimized out>) at
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:898
#21 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>,
s=<value optimized out>) at
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:1102
#22 0x00007fdd375e1840 in main (argc=1, argv=0x7fff7d2a5858) at
/usr/src/debug/httpd-2.2.15/server/main.c:740
(gdb)
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
