Issue #1095 has been updated by Ricky Zhou.
Hi, I just tried to reproduce this on another F11 machine. At first, I wasn't able to, but after looking at the differences, I noticed that SELinux was disabled on the machine that did not exhibit the behavior. When I enabled SELinux (permissive, not enforcing), I was able to reproduce the issue. Here are the steps that I took to do so: <pre> yum install httpd mod_ssl mysql-server ruby ruby-augeas ruby-shadow which ruby-RRDtool rubygem-rails rubygem-mongrel ruby-mysql facter rpm -Uvh \ http://tmz.fedorapeople.org/repo/puppet/fedora/11/x86_64/puppet-0.25.0-0.2.beta2.fc11.noarch.rpm \ http://tmz.fedorapeople.org/repo/puppet/fedora/11/x86_64/puppet-server-0.25.0-0.2.beta2.fc11.noarch.rpm /etc/init.d/mysqld start mysql -u root create database puppet; grant all privileges on puppet.* to pup...@localhost identified by 'puppet'; wget -O /etc/puppet/puppet.conf http://ricky.fedorapeople.org/puppet.conf echo 'PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )' > /etc/sysconfig/puppetmaster puppetca --generate publictest1.fedoraproject.org wget -O /etc/httpd/conf.d/puppetmaster-mongrel.conf http://ricky.fedorapeople.org/puppetmaster-mongrel.conf echo 'notice("This is a test")' > /etc/puppet/manifests/site.pp /etc/init.d/puppetmaster start /etc/init.d/httpd start puppetd -t </pre> ---------------------------------------- Bug #1095: Puppetmaster leaving half-open connections http://projects.reductivelabs.com/issues/1095 Author: Frank Sweetser Status: Re-opened Priority: High Assigned to: Luke Kanies Category: network Target version: 0.25.0 Complexity: Medium Patch: None Affected version: 0.25.0beta2 Keywords: After a period of time ranging from a few hours to several days, puppetmaster begins leaving half open TCP connections in a CLOSE_WAIT state. It usually seems to happen to connections from clients, though at least once I've seen it hit the database connection (MySQL). Here's an example: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetd 13420 root 7u IPv4 48150014 TCP lorien.wpi.edu:52225->lorien.wpi.edu:8140 (ESTABLISHED) puppetmas 13744 puppet 10u IPv4 47981997 TCP *:8140 (LISTEN) puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> Once puppetmaster starts leaking sockets like this, it seems unable to answer any new requests. In this example, you can see that the puppet client on the local machine (lorien) has opened a connection to puppetmaster, but puppetmaster has not responded. None of the log files on either master or client show that any progress has been made. Sending a HUP to the server generates "Restarting" and "Shutting down" messages in syslog, but it never restarts. lsof shows that there are puppetmaster processes hanging around keeping the original set of half open sockets open, but nothing is listening for new connections anymore: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> A full restart of puppetmaster appears to be the only way to get things flowing again. This is on 0.24.1 plus the patch from ticket 959. Let me know what other debugging info you'd like me to gather up. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
