Issue #1689 has been updated by aionica.
ohadlevy wrote: > jamtur01 wrote: > > I don't see know this could be a Puppet issue? I have never seen this - > > nor can I replicate it. Has it re-occurred? > > I've seen it before, usually when connecting to a network where puppet > resolves but its not the correct certificate (e.g. were signed by another > puppetmaster) > > It took me a while to see that puppet is abusing my network (learned it only > when my home router cpu increased because of so many dns requests) Since the initial setup i had two occurrences that i know of. I'm 100% sure the certificates were ok, we have only one puppetmaster and we already have a procedure for installation of new nodes so things should be the same on all nodes. Since i opened the ticket i didn't notice new occurences but as said it's hard to spot them. I don't want to leave query log enabled on the dns server as last time it filled up all of the disk space (30GB). On the other side if it happens one way to spot it is by the size of the log files (we do network logging so all logs are in one place to check). If it's not a puppet issue than what could it be ? From my point of view i'm sure it's not the dns system at least as queries are answered by the servers but something happens on the client. It guess it may be Puppet or some component from Ruby. In the mean time i started to add more clients to the setup (and a second caching dns server to cope with possible problems). I'll get back when i have news. ---------------------------------------- Bug #1689: Hundreds of log messages a second because of dns resolution http://projects.reductivelabs.com/issues/show/1689 Author: aionica Status: Needs more information Priority: Normal Assigned to: Category: unknown Target version: Complexity: Unknown Affected version: 0.24.5 Keywords: Hello, I have a puppet installation with one master and around 20 clients. The problem appeared until now twice, and it seems to be related to dns resolution and it's generating around 700 log messages / second in syslog. In 80 minutes it generated 3070322 log lines which is huge. The problem went away by itself as it appeared (i didn't do anything to fix it) and i discovered it because of the huge log size. OS/Distro running on master and clients is Debian Etch, kernel 2.6.18-6, Facter version 1.5.1 , Puppetd 0.24.5 (Debian package versions are: 1.5.1-0.1 for Facter and 0.24.5-2 for Puppet, both taken from testing/Lenny branch) and Ruby version is 1.8.2 (Debian Etch/Stable package). I pasted below the log entries taken from syslog. <pre> Oct 21 10:32:18 localhost puppetd[17738]: Reopening log files Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known </pre> additional 3070300 lines with the same thing <pre> Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:30 localhost puppetd[18465]: Creating default schedules Oct 21 11:52:30 localhost puppetd[18465]: Failed to load library 'ldap' for feature 'ldap' Oct 21 11:52:30 localhost puppetd[18465]: (/Settings[/etc/puppet/puppetd.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]) Autorequiring File[/etc/puppet] </pre> The weird thing is that in the end it got the manifest file and ran it. I doubt that dns wasn't working for 80 minutes withing the network as we would have had a lot of rings and bells announcing this but even if the dns was dead the puppetd daemon shouldn't do this. I mentioned above that we had this issue two times in two weeks since running puppet. The first time we had it i also (by chance) had the query log enabled on the dns server and i saw thousands of queries for the puppetmaster server, each query was answerd. I fixed this then by killing puppetd and starting it again on the client. The second time the issue appeard(the issue described above) the query log was disabled. I will also paste the client and server config below, the only thing changed is the domain name , to foo.bar.com : Client config(puppetd.conf): <pre> [puppetd] logdir=/var/log/puppet vardir=/var/lib/puppet rundir=/var/run/puppet server = puppetmaster.foo.bar.com usecacheonfailure = false runinterval = 900 report = true listen = true </pre> Puppetmaster config options are passed as parameters to puppetmasterd, the parameters are: <pre> --templatedir=/etc/puppet/manifests/templates [EMAIL PROTECTED] --reports=tagmail,store,rrdgraph --rrdgraph --reportdir=/var/www/reports/puppetreports --rrddir=/var/www/reports/rrd --certname=puppetmaster.foo.bar.com --logdir=/var/log/puppet --vardir=/var/lib/puppet --rundir=/var/run/puppet --rrdinterval=900 --runinterval=900 </pre> ---------------------------------------- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
