So, we got access back. It turned out to be a malformed fact causing the problem. The particular fact was using a ruby exec to get lvm freespace, using chomp on the result without checking wether there was a result. Since the x7 server didn't have lvm installed to begin with, that led to problems.
So, just to make sure I understand this correctly: in this case just removing all the puppet code didn't help, since the mere existence of the module with the custom fact in our module path made the fact execute on the agent, right? cheers, Walter On Wed, Apr 25, 2012 at 01:18, Michael Baydoun <indymicha...@gmail.com> wrote: > Maybe you could use puppet to configure send to a remote syslog. If that > works, you could then add your problem module back, and get visibility to > exactly what the client is doing, and any errors it is seeing. > > Or make some small modification to the existing log file on the remote > server, with filebucketing to your master, and use that to retrieve the > puppet log from the remote > > > On Tue, Apr 24, 2012 at 9:09 AM, jcbollinger <john.bollin...@stjude.org> > wrote: >> >> >> >> On Apr 23, 9:43 pm, Walter Heck <walterh...@gmail.com> wrote: >> > Hi all, >> > >> > in an unfortunate incident, I managed to lock myself out of a client's >> > server. Basically an openssh module that by default disabled remote >> > root logins did that on a server that was only accessed by remote root >> > login (no other use raccounts present on that server). Unfortunately, >> > this is a colocated server and next trip to the dc is scheduled for >> > next thursday. No KVM over IP, no remote hands, pretty much the ideal >> > situation :P. The Xen server is still running, and so are the domU's >> > on it, but this is less then ideal. If any of the domU's goes down, >> > there's nothing we can do :) >> > >> > Now, the puppet agent is running every 30 minutes, but something seems >> > to make it not execute the catalog. I have set the puppetmaster to >> > debug in order to see what's happening, but I can't figure it out. >> > Here's a gist of the puppet master log:https://gist.github.com/2475554 >> > >> > x7 is the offending server, x6 has exactly the same puppet definition. >> > Can anyone tell me why the log for x7 just stops, with no error or >> > nothing? What does that indicate is happening on x7? Any help is much >> > appreciated :) >> >> >> In the excerpt you posted, it looks like x7 is getting a cached >> catalog, whereas x6's catalog needed to be recompiled. The fact that >> their manifests are the same is not inconsistent with that. Perhaps >> that's why you don't see more. >> >> Alternatively, most of the log lines pertaining to x7 appear to show >> it downloading plugins -- maybe you have a hung client, but successive >> cron-initiated runs are performing plugin sync and fact gathering >> before that stops them. >> >> Since it looks like the client is still (plugin)syncing, however, that >> may be enough of an opening for you to break the server back open. >> You could try sending it a custom fact that has whatever clever side >> effect you like. I'm not certain whether facts are evaluated with >> privilege, but you should at least be able to collect information and >> write it to your share. >> >> >> John >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. -- Walter Heck -- follow @walterheck on twitter to see what I'm up to! -- Check out my new startup: Server Monitoring as a Service @ http://tribily.com Follow @tribily on Twitter and/or 'Like' our Facebook page at http://www.facebook.com/tribily -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
