Issue #11257 has been updated by Josh Cooper. Status changed from Unreviewed to Investigating
This is a sign that the apache server has received a TCP FIN from the client, but the thread handling the connection in the apache server has not called close (and the thread dump shows that it's stuck in read). ---------------------------------------- Bug #11257: Puppetmaster accumulates CLOSE_WAIT connections in apache server https://projects.puppetlabs.com/issues/11257 Author: Steven Seed Status: Investigating Priority: Normal Assignee: Category: passenger Target version: Affected Puppet version: 2.7.5 Keywords: close_wait, passenger, apache Branch: I have two nearly identically configured puppetmasters situated behind a load balancer. Both are running puppet-server-2.7.5-1 on RHEL 6.1. Periodically one of the two starts to accumulate CLOSE_WAIT connections on port 8140 through the apache service until the maxclients count in apache is reached (in my case it is set to 300). Once this happens, my server will no long accept connections. The client connection have been closed, but the server keeps them open in the CLOSE_WAIT state for some reason. Nothing is logged on either the client or server side after this point. So far this has happened twice over the last couple of months requiring me to restart the apache service. Here are the relevant apache settings: <pre> <IfModule worker.c> StartServers 4 MaxClients 300 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule> </pre> The puppetmaster is run through Passenger. Here are the relevant sections of my rack.conf file: <pre> LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9/ext/apache2/mod_passenger.so PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.9 PassengerRuby /usr/bin/ruby # tune passenger performance PassengerHighPerformance on PassengerMaxPoolSize 24 PassengerPoolIdleTime 300 PassengerMaxRequests 10000 PassengerStatThrottleRate 120 RackAutoDetect Off RailsAutoDetect Off </pre> Last line in the http error_log file prior to hitting the max connections... <pre> [Wed Dec 07 08:22:34 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting </pre> I attached a debugger to one of the httpd processes in the CLOSE_WAIT state. Here are is the back trace: <pre> (gdb) bt #0 0x00007fdd35ea84c0 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007fdd299a8a8f in oxt::syscalls::read (fd=19, buf=0x7fff7d2a228e, count=2) at ext/oxt/system_calls.cpp:111 #2 0x00007fdd2999ac40 in Passenger::readExact (fd=19, buf=0x7fff7d2a228e, size=2, timeout=0x0) at ext/common/Utils/IOUtils.cpp:613 #3 0x00007fdd2995eb0d in Passenger::readUint16 (fd=19, output=@0x7fff7d2a233e, timeout=0x0) at ext/common/Utils/MessageIO.h:124 #4 0x00007fdd29982078 in Passenger::readArrayMessage<std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > ( fd=19, output=std::vector of length 0, capacity 0, timeout=0x0) at ext/common/Utils/MessageIO.h:239 #5 0x00007fdd2995ebf1 in Passenger::readArrayMessage (fd=19, timeout=0x0) at ext/common/Utils/MessageIO.h:282 #6 0x00007fdd2995f236 in Passenger::MessageChannel::read (this=0x7fdd38c48310, args=std::vector of length 0, capacity 0) at ext/common/MessageChannel.h:315 #7 0x00007fdd29973d93 in Passenger::ApplicationPool::Client::sendGetCommand (this=0x7fdd386b3210, options=..., reply=std::vector of length 0, capacity 0) at ext/common/ApplicationPool/Client.h:367 #8 0x00007fdd29976185 in Passenger::ApplicationPool::Client::get (this=0x7fdd386b3210, options=...) at ext/common/ApplicationPool/Client.h:708 #9 0x00007fdd299791b5 in Hooks::getSession (this=0x7fdd38a1ab00, options=...) at ext/apache2/Hooks.cpp:300 #10 0x00007fdd2997ad2e in Hooks::handleRequest (this=0x7fdd38a1ab00, r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:670 #11 0x00007fdd29981318 in Hooks::handleRequestWhenInHighPerformanceMode (this=0x7fdd38a1ab00, r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:1624 #12 0x00007fdd2995bbd7 in handle_request_when_in_high_performance_mode (r=0x7fdd38c59038) at ext/apache2/Hooks.cpp:1786 #13 0x00007fdd375f5980 in ap_run_handler (r=0x7fdd38c59038) at /usr/src/debug/httpd-2.2.15/server/config.c:158 #14 0x00007fdd375f923e in ap_invoke_handler (r=0x7fdd38c59038) at /usr/src/debug/httpd-2.2.15/server/config.c:376 #15 0x00007fdd37604880 in ap_process_request (r=0x7fdd38c59038) at /usr/src/debug/httpd-2.2.15/modules/http/http_request.c:282 #16 0x00007fdd37601748 in ap_process_http_connection (c=0x7fdd38a26038) at /usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190 #17 0x00007fdd375fd488 in ap_run_process_connection (c=0x7fdd38a26038) at /usr/src/debug/httpd-2.2.15/server/connection.c:43 #18 0x00007fdd37609657 in child_main (child_num_arg=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:667 #19 0x00007fdd3760996a in make_child (s=0x7fdd38523860, slot=254) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:763 #20 0x00007fdd3760a5ec in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:898 #21 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at /usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:1102 #22 0x00007fdd375e1840 in main (argc=1, argv=0x7fff7d2a5858) at /usr/src/debug/httpd-2.2.15/server/main.c:740 (gdb) </pre> -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
