On Feb 10, 2009, at 4:25 AM, Keith Edmunds wrote: > > I'm just starting a roll out of Puppet and I'm seeing a problem on > maybe > 25% of client nodes. The symptoms are that the clients stop > updating. In > the Puppetmaster log, I'm seeing things like: > > Feb 9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in > 0.05 seconds > Feb 9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in > 0.05 seconds > Feb 9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in > 1.83 seconds > Feb 9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in > 0.91 seconds > > These are all for the same client; everything appears normal until > 21:41, > then no more checks from the client (it's now 10:17 on Feb 10). > > On the client, I tried running puppetd manually: > > # puppetd -t > notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping > catalog run > > A look at the lock file: > > # ls -l /var/lib/puppet/state/puppetdlock > -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/ > puppetdlock > > ...shows that it was probably created at the next run after the last > one > logged on the Puppetmaster (above). > > Looking at the lock file: > > # echo $(cat /var/lib/puppet/state/puppetdlock) > 32400 > # ps -fp 32400 > UID PID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ? 00:01:41 ruby /usr/sbin/ > puppetd -w 5 > > ...shows that the puppetd is still running. > > Why would the lock file be created and not subsequently deleted? > > If it helps, it is likely that the Puppetmaster was very busy at that > time, but even so I would expect the client to deal with that > graciously. > > Maybe related, maybe not: I can't stop puppetd in the usual way: > > # /etc/init.d/puppet stop > Stopping puppet configuration management tool. > # ps -fp 32400 > UID PID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ? 00:01:41 ruby /usr/sbin/ > puppetd -w 5 > > If I 'kill -9' the puppetd process, remove the /var/run/puppetd.pid > file > and remove the lock file, I can restart puppetd and it runs OK for a > while, but eventually the puppetdlock file causes this problem again. > > Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch. > > Grateful for any suggestions / pointers / etc.
I haven't seen this for absolute ages, so whatever used to cause it is unlikely to be the problem now. Is there maybe a signal being sent to puppetd? If not, any chance you can get a stack trace from the clients? You'd have to leave a client running in the foreground with stdout redirected. Otherwise.... I've no idea, because you're the first to run into it that I know of. Note that you should be able to run 'puppetd --enable' to remove that stale lock file. -- One of the keys to happiness is a bad memory. -- Rita Mae Brown --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
