Issue #1095 has been updated by Ricky Zhou.
Here's the output of puppetmasterd --debug --trace --no-daemonize --servertype=mongrel --masterport=18140 --color false on one of the machines where I was able to reproduce. I ran this command, then did a puppetd -t, which timed out. I ended up having to kill -9 the process when it did not respond to ctrl-c. Unfortunately, nothing seems too out of the ordinary in the output. <pre> debug: /File[/var/lib/puppet/ssl/crl.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/run/puppet]/seluser: Found seluser default 'system_u' for /var/run/puppet debug: /File[/var/run/puppet]/selrole: Found selrole default 'object_r' for /var/run/puppet debug: /File[/var/run/puppet]/seltype: Found seltype default 'var_run_t' for /var/run/puppet debug: /File[/var/run/puppet]/selrange: Found selrange default 's0' for /var/run/puppet debug: /File[/var/lib/puppet/ssl/public_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/public_keys debug: /File[/etc/puppet]/seluser: Found seluser default 'system_u' for /etc/puppet debug: /File[/etc/puppet]/selrole: Found selrole default 'object_r' for /etc/puppet debug: /File[/etc/puppet]/seltype: Found seltype default 'etc_t' for /etc/puppet debug: /File[/etc/puppet]/selrange: Found selrange default 's0' for /etc/puppet debug: /File[/var/lib/puppet/rrd]/seluser: Found seluser default 'system_u' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/selrole: Found selrole default 'object_r' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/selrange: Found selrange default 's0' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/ssl/certs]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/facts/]/seluser: Found seluser default 'system_u' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/selrole: Found selrole default 'object_r' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/selrange: Found selrange default 's0' for /var/lib/puppet/facts debug: /File[/var/log/puppet]/seluser: Found seluser default 'system_u' for /var/log/puppet debug: /File[/var/log/puppet]/selrole: Found selrole default 'object_r' for /var/log/puppet debug: /File[/var/log/puppet]/seltype: Found seltype default 'var_log_t' for /var/log/puppet debug: /File[/var/log/puppet]/selrange: Found selrange default 's0' for /var/log/puppet debug: /File[/var/lib/puppet/reports]/seluser: Found seluser default 'system_u' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/selrole: Found selrole default 'object_r' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/selrange: Found selrange default 's0' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/lib]/seluser: Found seluser default 'system_u' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/selrole: Found selrole default 'object_r' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/selrange: Found selrange default 's0' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/bucket]/seluser: Found seluser default 'system_u' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/selrole: Found selrole default 'object_r' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/selrange: Found selrange default 's0' for /var/lib/puppet/bucket debug: /File[/etc/puppet/fileserver.conf]/seluser: Found seluser default 'system_u' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/selrole: Found selrole default 'object_r' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/seltype: Found seltype default 'etc_t' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/selrange: Found selrange default 's0' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/puppet.conf]/seluser: Found seluser default 'system_u' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/selrole: Found selrole default 'object_r' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/seltype: Found seltype default 'etc_t' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/selrange: Found selrange default 's0' for /etc/puppet/puppet.conf debug: /File[/var/lib/puppet/ssl/private]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private debug: /File[/etc/puppet/manifests]/seluser: Found seluser default 'system_u' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/selrole: Found selrole default 'object_r' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/seltype: Found seltype default 'etc_t' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/selrange: Found selrange default 's0' for /etc/puppet/manifests debug: /File[/var/lib/puppet/ssl/certificate_requests]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certificate_requests debug: /File[/var/lib/puppet/ssl/certificate_requests]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certificate_requests debug: /File[/var/lib/puppet/ssl/certificate_requests]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certificate_requests debug: /File[/var/lib/puppet/ssl/certificate_requests]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certificate_requests debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/yaml debug: /File[/etc/puppet/manifests/site.pp]/seluser: Found seluser default 'system_u' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/selrole: Found selrole default 'object_r' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/seltype: Found seltype default 'etc_t' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/selrange: Found selrange default 's0' for /etc/puppet/manifests/site.pp debug: /File[/var/lib/puppet]/seluser: Found seluser default 'system_u' for /var/lib/puppet debug: /File[/var/lib/puppet]/selrole: Found selrole default 'object_r' for /var/lib/puppet debug: /File[/var/lib/puppet]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet debug: /File[/var/lib/puppet]/selrange: Found selrange default 's0' for /var/lib/puppet debug: /File[/var/log/puppet/masterhttp.log]/seluser: Found seluser default 'system_u' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/selrole: Found selrole default 'object_r' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/seltype: Found seltype default 'var_log_t' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/selrange: Found selrange default 's0' for /var/log/puppet/masterhttp.log debug: /File[/var/lib/puppet/ssl/private_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/state]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/selrange: Found selrange default 's0' for /var/lib/puppet/state debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: Failed to load library 'ldap' for feature 'ldap' debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet] debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet] debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet] debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] debug: /File[/etc/puppet/manifests/site.pp]: Autorequiring File[/etc/puppet/manifests] debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet] debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring File[/var/log/puppet] debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] debug: /File[/var/lib/puppet/rrd]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/etc/puppet/fileserver.conf]: Autorequiring File[/etc/puppet] debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] debug: Finishing transaction 70360911177640 with 0 changes debug: /File[/var/lib/puppet/ssl/ca/signed]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/private]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca/requests]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/serial]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/private]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/signed]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]: Autorequiring File[/var/lib/puppet/ssl/ca/private] debug: /File[/var/lib/puppet/ssl/ca/requests]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: Finishing transaction 70360910955680 with 0 changes debug: Using cached certificate for ca debug: Using cached certificate for ca debug: Using cached certificate for publictest1.fedoraproject.org notice: Starting Puppet server version 0.25.0 debug: Mongrel client debugging enabled. [$mongrel_debug_client = true]. debug: Finishing transaction 70360896283500 with 0 changes debug: No modules mount given; autocreating with default permissions debug: No plugins mount given; autocreating with default permissions debug: Creating interpreter debug: Finishing transaction 70360896268760 with 0 changes warning: Inserting default '~ ^/catalog/([^/]+)$'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/file'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate_revocation_list/ca'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/report'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate/ca'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate/'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate_request'(non-auth) acl because none were found in 'no auth.conf file configured' </pre> The Apache error logs contained a bunch of lines like: <pre> [Wed Jul 01 04:06:56 2009] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:18140 (127.0.0.1) failed [Wed Jul 01 04:06:56 2009] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1) </pre> ---------------------------------------- Bug #1095: Puppetmaster leaving half-open connections http://projects.reductivelabs.com/issues/1095 Author: Frank Sweetser Status: Re-opened Priority: High Assigned to: Luke Kanies Category: network Target version: 0.25.0 Complexity: Medium Patch: None Affected version: 0.25.0beta2 Keywords: After a period of time ranging from a few hours to several days, puppetmaster begins leaving half open TCP connections in a CLOSE_WAIT state. It usually seems to happen to connections from clients, though at least once I've seen it hit the database connection (MySQL). Here's an example: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetd 13420 root 7u IPv4 48150014 TCP lorien.wpi.edu:52225->lorien.wpi.edu:8140 (ESTABLISHED) puppetmas 13744 puppet 10u IPv4 47981997 TCP *:8140 (LISTEN) puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> Once puppetmaster starts leaking sockets like this, it seems unable to answer any new requests. In this example, you can see that the puppet client on the local machine (lorien) has opened a connection to puppetmaster, but puppetmaster has not responded. None of the log files on either master or client show that any progress has been made. Sending a HUP to the server generates "Restarting" and "Shutting down" messages in syslog, but it never restarts. lsof shows that there are puppetmaster processes hanging around keeping the original set of half open sockets open, but nothing is listening for new connections anymore: <pre> [r...@lorien ~]# lsof -i | grep 8140 puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT) </pre> A full restart of puppetmaster appears to be the only way to get things flowing again. This is on 0.24.1 plus the patch from ticket 959. Let me know what other debugging info you'd like me to gather up. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
