Kurt wrote:
> Hello list,
>
> In our company we use puppet to configure our ec2 instances.
> we start our servers with fresh instances debian lenny, some small and
> some large.
>
> we install puppet like this:
> apt-get -y --force-yes install ruby1.8 rubygems1.8
> gem install puppet --include-dependencies --version 0.24.8 --no-rdoc --
> no-ri
> ln -s /var/lib/gems/1.8/bin/puppetd /usr/sbin/puppetd
> ln -s /var/lib/gems/1.8/bin/puppet /usr/sbin/puppet
> ln -s /var/lib/gems/1.8/bin/facter /usr/bin/facter
>
> and create a puppet client config file:
> echo "[puppetd]" > /etc/puppet/puppetd.conf
> echo "logdir=/var/log/puppet" >> /etc/puppet/puppetd.conf
> echo "vardir=/var/lib/puppet" >> /etc/puppet/puppetd.conf
> echo "rundir=/var/run" >> /etc/puppet/puppetd.conf
> echo "server=<%=env_puppetmaster_fqdn%>" >> /etc/puppet/puppetd.conf
> echo "masterport=<%=pm_apache2_lb_port%>" >> /etc/puppet/puppetd.conf
> echo "pluginsync=true" >> /etc/puppet/puppetd.conf
> echo "factpath = /var/lib/puppet/lib/facter" >> /etc/puppet/
> puppetd.conf
>
>
> and start a first puppetrun with
> /usr/sbin/puppetd --test -v --waitforcert 120
>
> The problem:
>
> sometimes (not always) the client "breaks", uses 100%CPU, runs
> manifest very very slowly and sometimes never finishes.
>
> Killing the Client after
> puppetd -v --test --debug shows:
>
> debug: //Node[default]/milter-sms/milter-sms::mimedefang/Runit_service
> [mimedefang-multiplexor]/File[run-logfile mimedefang-multiplexor]/
> checksum: Initializing checksum hash
> debug: //Node[default]/milter-sms/milter-sms::mimedefang/Runit_service
> [mimedefang-multiplexor]/File[run-logfile mimedefang-multiplexor]:
> Creating checksum {md5}d6a4b36fea3bf1f7cd0c6e70989ca8c7
> notice: Caught INT; shutting down
> debug: Signal caught here:
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `call'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 262:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `each'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/simple_graph.rb:
> 260:in `walk'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/pgraph.rb:
> 116:in `tree_from_vertex'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/pgraph.rb:37:in
> `dependencies'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 346:in `failed_dependencies?'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 619:in `skip?'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 219:in `eval_resource'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 310:in `evaluate'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:425:in
> `thinmark'
> debug: /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:424:in
> `thinmark'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 309:in `evaluate'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 303:in `collect'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/transaction.rb:
> 303:in `evaluate'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/node/catalog.rb:
> 124:in `apply'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/
> master.rb:256:in `run'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:180:in
> `benchmark'
> debug: /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/util.rb:179:in
> `benchmark'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/
> master.rb:255:in `run'
> debug: /usr/lib/ruby/1.8/sync.rb:230:in `synchronize'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/lib/puppet/network/client/
> master.rb:237:in `run'
> debug: /var/lib/gems/1.8/gems/puppet-0.24.8/bin/puppetd:417
> debug: /usr/sbin/puppetd:19:in `load'
> debug: /usr/sbin/puppetd:19
> notice: Shutting down
> debug: Storing state
> debug: Stored state in 0.37 seconds
> debug: Storing state
> debug: Stored state in 0.52 seconds
>
> The problem:
>
> sometimes this never happens on one ec2-instance.
> we can start new "fresh" instances and sometimes it always works with
> the same manifests file.
> we have different "roles" in our environment with different manifests/
> classes/modules and when we start a puppetrun on role1 and another on
> role2 it happens that role2 fails and role1 not. Starting the run
> again sometimes both fail, sometimes both are ok, sometimes they
> switch.
>
> puppet mostly "stops" when creating checksums of configuration files
> (template based or via fileserver-mount) and at the end of the
> puppetrun.
> Files which are checked at the beginning of the puppetrun (because
> this Files are first checked because of require/subscribe direcives)
> always gets done fast. Files which are checked at the end hangs on
> md5summing.
>
> The thing is that I cant reproduce this problem when I want to.
>
> Deleting /var/lib/puppet/* and running the client does not always
> helps.
>
> Anyone experiences the same problem sometimes or could give a hint
> what it could be?
>
> Many thanks,
>
> Kurt B.
>
> >
>
Hi Kurt, possibly some kind of weird clash between gems and packages,
which is fairly common in deb/ubuntu if you mix gems with os-native
dpkg. I have the .24.8 package from the pool ftp.ubuntu.com (analogous
to ftp.debian.com) which seems to work pretty well in my environment and
I don't really hit any 'strange' issues.
--
Joe McDonagh
Operations Engineer
www.colonfail.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
