Issue #1689 has been updated by luke.
bart wrote: > I think there might be two separate issues here, but to zoom in on the > hammering: I've also seen puppetd continously hammer the puppetmaster when > puppetd requests a new certificate from puppetmaster, but for whichever > reason doesn't get one. After a while the log got filled with thousands of > lines, and the server became unresponsive. I had to forcibly kill the puppet > client to get things back to normal. > > The reason this happened for me was because puppetd was run with '-w 0' as > one of the command line arguments, which is apparently what the Debian > package for Puppet does by default (see /etc/default/puppet.) Passing -w 0 > does not seem to cause puppetd to exit when it doesn't get a certificate on > the first try (which is the behaviour I would expect); instead it waits for > 0 seconds, and then tries again, effectively DoS'ing the server. > > I've only tested this with Puppet 0.24.5 and 0.24.6, so I don't know if that > behaviour still stands for 0.24.7. Looks like the behaviour is still the same in 0.24.7, but it's not in the master branch. Can you modify your client to break its loop if waitforcert < 1 and see if that affects the problem? ---------------------------------------- Bug #1689: Hundreds of log messages a second because of dns resolution http://projects.reductivelabs.com/issues/show/1689 Author: aionica Status: Needs more information Priority: Normal Assigned to: Category: unknown Target version: Complexity: Unknown Affected version: 0.24.5 Keywords: Hello, I have a puppet installation with one master and around 20 clients. The problem appeared until now twice, and it seems to be related to dns resolution and it's generating around 700 log messages / second in syslog. In 80 minutes it generated 3070322 log lines which is huge. The problem went away by itself as it appeared (i didn't do anything to fix it) and i discovered it because of the huge log size. OS/Distro running on master and clients is Debian Etch, kernel 2.6.18-6, Facter version 1.5.1 , Puppetd 0.24.5 (Debian package versions are: 1.5.1-0.1 for Facter and 0.24.5-2 for Puppet, both taken from testing/Lenny branch) and Ruby version is 1.8.2 (Debian Etch/Stable package). I pasted below the log entries taken from syslog. <pre> Oct 21 10:32:18 localhost puppetd[17738]: Reopening log files Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known </pre> additional 3070300 lines with the same thing <pre> Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:30 localhost puppetd[18465]: Creating default schedules Oct 21 11:52:30 localhost puppetd[18465]: Failed to load library 'ldap' for feature 'ldap' Oct 21 11:52:30 localhost puppetd[18465]: (/Settings[/etc/puppet/puppetd.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]) Autorequiring File[/etc/puppet] </pre> The weird thing is that in the end it got the manifest file and ran it. I doubt that dns wasn't working for 80 minutes withing the network as we would have had a lot of rings and bells announcing this but even if the dns was dead the puppetd daemon shouldn't do this. I mentioned above that we had this issue two times in two weeks since running puppet. The first time we had it i also (by chance) had the query log enabled on the dns server and i saw thousands of queries for the puppetmaster server, each query was answerd. I fixed this then by killing puppetd and starting it again on the client. The second time the issue appeard(the issue described above) the query log was disabled. I will also paste the client and server config below, the only thing changed is the domain name , to foo.bar.com : Client config(puppetd.conf): <pre> [puppetd] logdir=/var/log/puppet vardir=/var/lib/puppet rundir=/var/run/puppet server = puppetmaster.foo.bar.com usecacheonfailure = false runinterval = 900 report = true listen = true </pre> Puppetmaster config options are passed as parameters to puppetmasterd, the parameters are: <pre> --templatedir=/etc/puppet/manifests/templates [email protected] --reports=tagmail,store,rrdgraph --rrdgraph --reportdir=/var/www/reports/puppetreports --rrddir=/var/www/reports/rrd --certname=puppetmaster.foo.bar.com --logdir=/var/log/puppet --vardir=/var/lib/puppet --rundir=/var/run/puppet --rrdinterval=900 --runinterval=900 </pre> ---------------------------------------- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
