Issue #1095 has been updated by threetee. Affected version set to 0.22.1
I am running into the same problem with my puppet setup. I have two separate puppet installations, one with ~150 nodes, and the other with ~300. Both puppetmasters are running on CentOS 5.2, using WEBrick, and have storeconfigs enabled. I am using the C-based MySQL libraries (MySQL gem, installed via the ruby-mysql RPM). When puppetmaster hangs, I see inbound connections to puppetmaster start to stack up: # netstat -an | grep 8140 | grep SYN_RECV | wc -l 55 # netstat -an | grep 8140 | wc -l 67 However, it doesn't look like open filehandles are the problem here: # lsof -u puppet | wc -l 77 I do see multiple filehandles for the /var/log/puppet/rails.log and masterhttp.log files: # lsof -u puppet | grep rails.log | wc -l 10 # lsof -u puppet | grep masterhttp.log | wc -l 3 puppetmaster is running puppet 0.24.7, clients are running versions between 0.24.4 and 0.24.7. I'm going to try using Apache/Passenger to see if it alleviates the problem. Any other ideas? ---------------------------------------- Bug #1095: Puppetmaster leaving half-open connections http://projects.reductivelabs.com/issues/show/1095 Author: fs Status: Needs more information Priority: High Assigned to: luke Category: network Target version: 0.25.0 Complexity: Medium Patch: None Affected version: 0.22.1 Keywords: After a period of time ranging from a few hours to several days, puppetmaster begins leaving half open TCP connections in a CLOSE_WAIT state. It usually seems to happen to connections from clients, though at least once I've seen it hit the database connection (MySQL). Here's an example: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetd 13420 root 7u IPv4 48150014 TCP lorien.wpi.edu:52225->lorien.wpi.edu:8140 (ESTABLISHED) puppetmas 13744 puppet 10u IPv4 47981997 TCP *:8140 (LISTEN) puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> Once puppetmaster starts leaking sockets like this, it seems unable to answer any new requests. In this example, you can see that the puppet client on the local machine (lorien) has opened a connection to puppetmaster, but puppetmaster has not responded. None of the log files on either master or client show that any progress has been made. Sending a HUP to the server generates "Restarting" and "Shutting down" messages in syslog, but it never restarts. lsof shows that there are puppetmaster processes hanging around keeping the original set of half open sockets open, but nothing is listening for new connections anymore: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> A full restart of puppetmaster appears to be the only way to get things flowing again. This is on 0.24.1 plus the patch from ticket 959. Let me know what other debugging info you'd like me to gather up. ---------------------------------------- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
