Issue #1689 has been updated by aionica.
ohadlevy wrote: > Steps how I reproduce the problem: > > Get a working client (e.g. already run puppet once, got certificate etc) > against puppet dns alias > > changed network (from home to work) where puppet.domain point to another > server where the certificate does not work. > started puppet as a daemon and the following errors start to apper in syslog: > [...] > > cpu load jumps (on my machine to about 15-20%, and puppetd hammers the > puppetmaster and dns for querying..... Back with news. Today while installing a new client the problem appeared. It seems that it was triggered immediately after puppetd was launched for the first time (by the Debian scripts, as soon as aptitude finished installing the package) and before i had the chance to edit the config files. */var/log/aptitude* <pre> Aptitude 0.4.4: log report Fri, Oct 31 2008 08:25:36 +0100 IMPORTANT: this log only lists intended actions; actions which fail due to dpkg problems may not be completed. Will install 13 packages, and remove 0 packages. 12.2MB of disk space will be used =============================================================================== [INSTALL, DEPENDENCIES] facter [INSTALL, DEPENDENCIES] irb1.8 [INSTALL, DEPENDENCIES] libopenssl-ruby [INSTALL, DEPENDENCIES] libopenssl-ruby1.8 [INSTALL, DEPENDENCIES] libreadline-ruby1.8 [INSTALL, DEPENDENCIES] libruby1.8 [INSTALL, DEPENDENCIES] libshadow-ruby1.8 [INSTALL, DEPENDENCIES] libxmlrpc-ruby [INSTALL, DEPENDENCIES] rdoc [INSTALL, DEPENDENCIES] rdoc1.8 [INSTALL, DEPENDENCIES] ruby [INSTALL, DEPENDENCIES] ruby1.8 [HOLD] libpq4 [HOLD] linux-image-2.6.18-6-686 [HOLD] linux-image-2.6.18-6-686-bigmem [HOLD] openssh-client [HOLD] openssh-server [HOLD] postfix [HOLD] smart [HOLD] ssh [HOLD] tzdata [INSTALL] puppet =============================================================================== </pre> */var/log/messages* <pre> Oct 31 08:25:42 node1 puppetd[11724]: Reopening log files Oct 31 08:25:42 node1 puppetd[11724]: Could not find server puppet: getaddrinfo: Name or service not known Oct 31 08:25:42 node1 puppetd[11724]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 31 08:25:42 node1 puppetd[11724]: Could not find server puppet: getaddrinfo: Name or service not known Oct 31 08:25:42 node1 puppetd[11724]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 31 08:25:42 node1 puppetd[11724]: Could not find server puppet: getaddrinfo: Name or service not known Oct 31 08:25:42 node1 puppetd[11724]: Could not request certificate: Certificate retrieval failed: Could not find server puppet </pre> * thousand of lines down the line of /var/log/messages* <pre> Oct 31 08:26:15 node1 puppetd[11724]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 31 08:26:15 node1 puppetd[11724]: Could not find server puppet: getaddrinfo: Name or service not known Oct 31 08:26:15 node1 puppetd[11724]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 31 08:26:21 node1 puppetd[11741]: Creating default schedules Oct 31 08:26:21 node1 puppetd[11741]: Failed to load library 'ldap' for feature 'ldap' Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/namespaceauth.conf]) Autorequiring File[/etc/puppet] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/ssl]) Autorequiring File[/etc/puppet] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/var/lib/puppet/state]) Autorequiring File[/var/lib/puppet] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/var/lib/puppet/lib]) Autorequiring File[/var/lib/puppet] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/certs]) Autorequiring File[/etc/puppet/ssl] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/public_keys]) Autorequiring File[/etc/puppet/ssl] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private]) Autorequiring File[/etc/puppet/ssl] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]) Autorequiring File[/etc/puppet] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private_keys]) Autorequiring File[/etc/puppet/ssl] Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/ssl]) File does not exist Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/ssl]) Changing ensure Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/ssl]) 1 change(s) Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[main]/File[/etc/puppet/ssl]/ensure) created Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private_keys]) File does not exist Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private_keys]) Changing ensure Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private_keys]) 1 change(s) Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private_keys]/ensure) created Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private]) File does not exist Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private]) Changing ensure Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private]) 1 change(s) Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/private]/ensure) created Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/public_keys]) File does not exist Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/public_keys]) Changing ensure Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/public_keys]) 1 change(s) Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/public_keys]/ensure) created Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/certs]) File does not exist Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/certs]) Changing ensure Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/certs]) 1 change(s) Oct 31 08:26:21 node1 puppetd[11741]: (/Settings[/etc/puppet/puppetd.conf]/Settings[ssl]/File[/etc/puppet/ssl/certs]/ensure) created Oct 31 08:26:21 node1 puppetd[11741]: Finishing transaction -606709178 with 5 changes Oct 31 08:26:21 node1 puppetd[11741]: Creating a new certificate request for node1.foo.bar.com Oct 31 08:26:21 node1 puppetd[11741]: Creating a new SSL key at /etc/puppet/ssl/private_keys/node1.foo.bar.com Oct 31 08:26:22 node1 puppetd[11741]: Calling puppetca.getcert Oct 31 08:26:22 node1 puppetd[11741]: Did not receive certificate Oct 31 08:26:22 node1 puppetd[11741]: Set to run 'one time'; exiting with no certificate </pre> So it seems that it's looking by default for the server with the name puppet , this is not the problem that the name does not exist but the fact that is hammering the dns server and the machine itself. Unfortunately for the other two occurrences i don't have the logfile any more to check and see if the problems were at the first run but i think the first run was the problem. I'm going to install more clients today and check the behave our . ---------------------------------------- Bug #1689: Hundreds of log messages a second because of dns resolution http://projects.reductivelabs.com/issues/show/1689 Author: aionica Status: Needs more information Priority: Normal Assigned to: Category: unknown Target version: Complexity: Unknown Affected version: 0.24.5 Keywords: Hello, I have a puppet installation with one master and around 20 clients. The problem appeared until now twice, and it seems to be related to dns resolution and it's generating around 700 log messages / second in syslog. In 80 minutes it generated 3070322 log lines which is huge. The problem went away by itself as it appeared (i didn't do anything to fix it) and i discovered it because of the huge log size. OS/Distro running on master and clients is Debian Etch, kernel 2.6.18-6, Facter version 1.5.1 , Puppetd 0.24.5 (Debian package versions are: 1.5.1-0.1 for Facter and 0.24.5-2 for Puppet, both taken from testing/Lenny branch) and Ruby version is 1.8.2 (Debian Etch/Stable package). I pasted below the log entries taken from syslog. <pre> Oct 21 10:32:18 localhost puppetd[17738]: Reopening log files Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 10:32:18 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 10:32:18 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known </pre> additional 3070300 lines with the same thing <pre> Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:22 localhost puppetd[17738]: Could not find server puppet: getaddrinfo: Name or service not known Oct 21 11:52:22 localhost puppetd[17738]: Could not request certificate: Certificate retrieval failed: Could not find server puppet Oct 21 11:52:30 localhost puppetd[18465]: Creating default schedules Oct 21 11:52:30 localhost puppetd[18465]: Failed to load library 'ldap' for feature 'ldap' Oct 21 11:52:30 localhost puppetd[18465]: (/Settings[/etc/puppet/puppetd.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]) Autorequiring File[/etc/puppet] </pre> The weird thing is that in the end it got the manifest file and ran it. I doubt that dns wasn't working for 80 minutes withing the network as we would have had a lot of rings and bells announcing this but even if the dns was dead the puppetd daemon shouldn't do this. I mentioned above that we had this issue two times in two weeks since running puppet. The first time we had it i also (by chance) had the query log enabled on the dns server and i saw thousands of queries for the puppetmaster server, each query was answerd. I fixed this then by killing puppetd and starting it again on the client. The second time the issue appeard(the issue described above) the query log was disabled. I will also paste the client and server config below, the only thing changed is the domain name , to foo.bar.com : Client config(puppetd.conf): <pre> [puppetd] logdir=/var/log/puppet vardir=/var/lib/puppet rundir=/var/run/puppet server = puppetmaster.foo.bar.com usecacheonfailure = false runinterval = 900 report = true listen = true </pre> Puppetmaster config options are passed as parameters to puppetmasterd, the parameters are: <pre> --templatedir=/etc/puppet/manifests/templates [EMAIL PROTECTED] --reports=tagmail,store,rrdgraph --rrdgraph --reportdir=/var/www/reports/puppetreports --rrddir=/var/www/reports/rrd --certname=puppetmaster.foo.bar.com --logdir=/var/log/puppet --vardir=/var/lib/puppet --rundir=/var/run/puppet --rrdinterval=900 --runinterval=900 </pre> ---------------------------------------- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
