Issue #2089 has been updated by Will Roberts. Status changed from Closed to Re-opened
I see this issue using puppet 3.1.1. I currently have 4 nodes that are stuck in this state. On a broken box netstat shows a connection: netstat -tan | grep 8140 tcp 0 0 myhungbox:48431 mymaster:8140 ESTABLISHED But no matching connection exists on the master. ps axfh 3882 ? Ss 0:01 /usr/bin/ruby /usr/bin/puppet agent 22308 ? S 0:00 \_ puppet agent: applying configurat strace -p 22308 Process 22308 attached - interrupt to quit select(5, [4], NULL, NULL, NULL I lose about 1 node a day, so if there's something I can do to help debug this, please let me know. ---------------------------------------- Bug #2089: puppet client on nodes hangs after various networking glitches https://projects.puppetlabs.com/issues/2089#change-88858 * Author: Max Stepanov * Status: Re-opened * Priority: Normal * Assignee: Nigel Kersten * Category: plumbing * Target version: * Affected Puppet version: 0.24.7 * Keywords: * Branch: ---------------------------------------- sometimes i find puppet on nodes stuck. usually it is several nodes together. they hang there doing "nothing". i restart those in order to get them running again. it seems network glitches are responsible for this behavior. here is a quick trace(it's the same on all nodes): hey:~# ps aux| grep puppet root 12629 0.5 2.1 129820 88896 ? Ss 03:41 2:26 ruby /usr/sbin/puppetd -w 0 hey:~# strace -f -p 12629 Process 12629 attached - interrupt to quit select(12, [9 10], [], [], {0, 184000}) = 0 (Timeout) select(12, [9 10], [], [], {0, 1477}) = 0 (Timeout) select(12, [9 10], [], [], {0, 0}) = 0 (Timeout) rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 select(12, [9 10], [], [], {1, 999998}) = 0 (Timeout) select(12, [9 10], [], [], {0, 1706}) = 0 (Timeout) select(12, [9 10], [], [], {0, 0}) = 0 (Timeout) rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 select(12, [9 10], [], [], {1, 999997} <unfinished ...> Process 12629 detached hey:~# file /proc/12629/fd/{9,10} /proc/12629/fd/9: broken symbolic link to `socket:[93425425]' /proc/12629/fd/10: broken symbolic link to `socket:[93847992]' hey:~# lsof -n | egrep "(93425425|93847992)" ruby 12629 root 9u IPv4 93425425 TCP *:8139 (LISTEN) ruby 12629 root 10u IPv4 93847992 TCP localip:46803->puppetmaster:8140 (ESTABLISHED) hey:# file /proc/12629/fd/* 0: symbolic link to `/dev/null' 1: symbolic link to `/dev/null' 10: broken symbolic link to `socket:[93847992]' 2: symbolic link to `/dev/null' 3: broken symbolic link to `pipe:[93425407]' 4: broken symbolic link to `pipe:[93425407]' 5: broken symbolic link to `socket:[93425419]' 6: symbolic link to `/var/log/puppet/http.log' 7: symbolic link to `/var/log/puppet/http.log' 8: symbolic link to `/var/log/puppet/http.log' 9: broken symbolic link to `socket:[93425425]' i'm ready to provide more info(if requested) next time it happens. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
