I just have to acknowledge that I too am seeing random problems with puppet which have advanced to "persistent problems".
In the beginning it was fine, I installed ruby from sources, then facter and then puppet and it worked like a charm. Then, 2 months ago I noticed that puppet was just not functioning anymore. Around that time I did some minor library upgrade (x.xx.YY <-- only YY changed) to the systems but I did it WITH puppet and it was working OK then. But then it died. Usually when one upgrades libraries only a recompile is required. I did recompile ruby, no difference. Now if I try to run it, it only runs once, or not even once. Then it just stucks at some "select(..." system call and that is it. Both, puppetmasterd and puppetd. If I kill and start both (the master and the client) it maybe runs once, maybe. Now, I do not have issues with things breaking down. What bothers me is that I cannot analyze this problem. Despite all the --debugging and stracing I can't figure out what is wrong. And this, sadly, is turning me to other solutions. Why sadly? Because it worked before like a charm and I know it can work like a charm, but if a tool is unreliable it is useless. Now, if anyone (besides the original poster) is experiencing similar problems and/or (even better) has found the cause for this, please let me know. Best regards, bostjan On Aug 30, 5:56 pm, Kurt <[email protected]> wrote: > Hello list, > > In our company we use puppet to configure our ec2 instances. > we start our servers with fresh instances debian lenny, some small and > some large. > > we install puppet like this: > apt-get -y --force-yes install ruby1.8 rubygems1.8 > gem install puppet --include-dependencies --version 0.24.8 --no-rdoc -- > no-ri > ln -s /var/lib/gems/1.8/bin/puppetd /usr/sbin/puppetd > ln -s /var/lib/gems/1.8/bin/puppet /usr/sbin/puppet > ln -s /var/lib/gems/1.8/bin/facter /usr/bin/facter > > and create a puppet client config file: > echo "[puppetd]" > /etc/puppet/puppetd.conf > echo "logdir=/var/log/puppet" >> /etc/puppet/puppetd.conf > echo "vardir=/var/lib/puppet" >> /etc/puppet/puppetd.conf > echo "rundir=/var/run" >> /etc/puppet/puppetd.conf > echo "server=<%=env_puppetmaster_fqdn%>" >> /etc/puppet/puppetd.conf > echo "masterport=<%=pm_apache2_lb_port%>" >> /etc/puppet/puppetd.conf > echo "pluginsync=true" >> /etc/puppet/puppetd.conf > echo "factpath = /var/lib/puppet/lib/facter" >> /etc/puppet/ > puppetd.conf > > and start a first puppetrun with > /usr/sbin/puppetd --test -v --waitforcert 120 > > The problem: > > sometimes (not always) the client "breaks", uses 100%CPU, runs > manifest very very slowly and sometimes never finishes. > > Killing the Client after > puppetd -v --test --debug shows: > > debug: //Node[default]/milter-sms/milter-sms::mimedefang/Runit_service > [mimedefang-multiplexor]/File[run-logfile mimedefang-multiplexor]/ > checksum: Initializing checksum hash > debug: //Node[default]/milter-sms/milter-sms::mimedefang/Runit_service > [mimedefang-multiplexor]/File[run-logfile mimedefang-multiplexor]: > Creating checksum {md5}d6a4b36fea3bf1f7cd0c6e70989ca8c7 > notice: Caught INT; shutting down > debug: Signal caught here: > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `call' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 262:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `each' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb: > 260:in `walk' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/pgraph.rb: > 116:in `tree_from_vertex' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/pgraph.rb:37:in > `dependencies' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 346:in `failed_dependencies?' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 619:in `skip?' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 219:in `eval_resource' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 310:in `evaluate' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:425:in > `thinmark' > debug: /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:424:in > `thinmark' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 309:in `evaluate' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 303:in `collect' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb: > 303:in `evaluate' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/node/catalog.rb: > 124:in `apply' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/ > master.rb:256:in `run' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:180:in > `benchmark' > debug: /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:179:in > `benchmark' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/ > master.rb:255:in `run' > debug: /usr/lib/ruby/1.8/sync.rb:230:in `synchronize' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/ > master.rb:237:in `run' > debug: /var/lib/gems/1.8/gems/puppet-0.24.8/bin/puppetd:417 > debug: /usr/sbin/puppetd:19:in `load' > debug: /usr/sbin/puppetd:19 > notice: Shutting down > debug: Storing state > debug: Stored state in 0.37 seconds > debug: Storing state > debug: Stored state in 0.52 seconds > > The problem: > > sometimes this never happens on one ec2-instance. > we can start new "fresh" instances and sometimes it always works with > the same manifests file. > we have different "roles" in our environment with different manifests/ > classes/modules and when we start a puppetrun on role1 and another on > role2 it happens that role2 fails and role1 not. Starting the run > again sometimes both fail, sometimes both are ok, sometimes they > switch. > > puppet mostly "stops" when creating checksums of configuration files > (template based or via fileserver-mount) and at the end of the > puppetrun. > Files which are checked at the beginning of the puppetrun (because > this Files are first checked because of require/subscribe direcives) > always gets done fast. Files which are checked at the end hangs on > md5summing. > > The thing is that I cant reproduce this problem when I want to. > > Deleting /var/lib/puppet/* and running the client does not always > helps. > > Anyone experiences the same problem sometimes or could give a hint > what it could be? > > Many thanks, > > Kurt B. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
