Issue #2089 has been updated by Wouter D'Haeseleer.
Actually the agent seems to HANG if this event has happend: ay 14 20:57:48 xen-7044-91 puppet-agent[2710]: Could not send report: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <html><head> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <title>500 Internal Server Error</title> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: </head><body> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <h1>Internal Server Error</h1> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <p>The server encountered an internal error or May 14 20:57:48 xen-7044-91 puppet-agent[2710]: misconfiguration and was unable to complete May 14 20:57:48 xen-7044-91 puppet-agent[2710]: your request.</p> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <p>Please contact the server administrator, May 14 20:57:48 xen-7044-91 puppet-agent[2710]: [no address given] and inform them of the time the error occurred, May 14 20:57:48 xen-7044-91 puppet-agent[2710]: and anything you might have done that may have May 14 20:57:48 xen-7044-91 puppet-agent[2710]: caused the error.</p> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <p>More information about this error may be available May 14 20:57:48 xen-7044-91 puppet-agent[2710]: in the server error log.</p> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <hr> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: <address>Apache/2.2.16 (Debian) Server at puppet Port 8140</address> May 14 20:57:48 xen-7044-91 puppet-agent[2710]: </body></html> May 14 21:01:01 xen-7044-91 puppet-agent[2710]: Could not retrieve catalog from remote server: Error 400 on SERVER: No space left on device May 14 21:01:01 xen-7044-91 puppet-agent[2710]: Using cached catalog ---------------------------------------- Bug #2089: puppet client on nodes hangs after various networking glitches https://projects.puppetlabs.com/issues/2089#change-91031 * Author: Max Stepanov * Status: Re-opened * Priority: Normal * Assignee: Nigel Kersten * Category: plumbing * Target version: * Affected Puppet version: 0.24.7 * Keywords: * Branch: ---------------------------------------- sometimes i find puppet on nodes stuck. usually it is several nodes together. they hang there doing "nothing". i restart those in order to get them running again. it seems network glitches are responsible for this behavior. here is a quick trace(it's the same on all nodes): hey:~# ps aux| grep puppet root 12629 0.5 2.1 129820 88896 ? Ss 03:41 2:26 ruby /usr/sbin/puppetd -w 0 hey:~# strace -f -p 12629 Process 12629 attached - interrupt to quit select(12, [9 10], [], [], {0, 184000}) = 0 (Timeout) select(12, [9 10], [], [], {0, 1477}) = 0 (Timeout) select(12, [9 10], [], [], {0, 0}) = 0 (Timeout) rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 select(12, [9 10], [], [], {1, 999998}) = 0 (Timeout) select(12, [9 10], [], [], {0, 1706}) = 0 (Timeout) select(12, [9 10], [], [], {0, 0}) = 0 (Timeout) rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 select(12, [9 10], [], [], {1, 999997} <unfinished ...> Process 12629 detached hey:~# file /proc/12629/fd/{9,10} /proc/12629/fd/9: broken symbolic link to `socket:[93425425]' /proc/12629/fd/10: broken symbolic link to `socket:[93847992]' hey:~# lsof -n | egrep "(93425425|93847992)" ruby 12629 root 9u IPv4 93425425 TCP *:8139 (LISTEN) ruby 12629 root 10u IPv4 93847992 TCP localip:46803->puppetmaster:8140 (ESTABLISHED) hey:# file /proc/12629/fd/* 0: symbolic link to `/dev/null' 1: symbolic link to `/dev/null' 10: broken symbolic link to `socket:[93847992]' 2: symbolic link to `/dev/null' 3: broken symbolic link to `pipe:[93425407]' 4: broken symbolic link to `pipe:[93425407]' 5: broken symbolic link to `socket:[93425419]' 6: symbolic link to `/var/log/puppet/http.log' 7: symbolic link to `/var/log/puppet/http.log' 8: symbolic link to `/var/log/puppet/http.log' 9: broken symbolic link to `socket:[93425425]' i'm ready to provide more info(if requested) next time it happens. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
