On Tue, Dec 11, 2012 at 10:21 AM, Roman Shaposhnik <[email protected]>wrote:
> Andy, Dominic, > > thanks a million for replying so quickly! > > On Tue, Dec 11, 2012 at 9:53 AM, Andy Parker <[email protected]> wrote: > > Have you been able to come up with a smaller catalog that reproduces the > > problem? Or something that doesn't include any sensitive information that > > you could send? > > There's nothing sensitive about that puppet code -- it is all part of > the Apache Bigtop project and is meant to be a portable way of > deploying Hadoop stacks (be gentle -- the code was written > prior to parameterized classes/hiera and it shows ;-)): > https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet > I will try to reproduce it with smaller catalog -- sure, but I don't think > I can make it small enough for the problem to be obvious :-( > > > I'd really like to know what the original error was. My suspicion is that > > because this is happening and various points it is either an out of > memory > > or related error. Applying Dominic's patch should expose the underlying > > error that is occurring. > > Here's a curious thing that happens when I do that. In the middle of the > run I start seeing the following stacktrace: > > That stacktrace is it trying to execute the start command for some service and failing. It should be providing a message like "Could not #{type} #{@resource.ref}: #{detail}". Do you get anything like at in the logs? > /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:60:in `fail' > /usr/lib/ruby/site_ruby/1.8/puppet/provider/service/base.rb:129:in > `texecute' > /usr/lib/ruby/site_ruby/1.8/puppet/provider/service/base.rb:141:in > `ucommand' > /usr/lib/ruby/site_ruby/1.8/puppet/provider/service/base.rb:82:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/type/service.rb:66:in `set_running' > /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:81:in `send' > /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:81:in `call_valuemethod' > /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:256:in `set' > /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:310:in `sync' > /usr/lib/ruby/site_ruby/1.8/puppet/type/service.rb:77:in `sync' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:114:in > `apply_parameter' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:56:in > `perform_changes' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:135:in > `evaluate' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:48:in `apply' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:113:in > `eval_children_and_apply_resource' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:91:in `eval_resource' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138:in `evaluate' > /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:403:in `thinmark' > /usr/lib/ruby/1.8/benchmark.rb:293:in `measure' > /usr/lib/ruby/1.8/benchmark.rb:307:in `realtime' > /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:402:in `thinmark' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:137:in `evaluate' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:130:in `each' > /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:130:in `evaluate' > /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:150:in `apply' > /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:120:in > `retrieve_and_apply_catalog' > /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:178:in `benchmark' > /usr/lib/ruby/1.8/benchmark.rb:293:in `measure' > /usr/lib/ruby/1.8/benchmark.rb:307:in `realtime' > /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:177:in `benchmark' > /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:119:in > `retrieve_and_apply_catalog' > /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:150:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:131:in `main' > /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:35:in `run_command' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:420:in `hook' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:411:in `exit_on_fail' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:62:in `execute' > /usr/bin/puppet:4 > > And at the very end of the run I see: > > /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:276:in `waitpid2': SIGHUP > (SignalException) > from /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:276:in `execute' > from /usr/lib/ruby/site_ruby/1.8/puppet/util/suidmanager.rb:150:in > `run_and_capture' > from > /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:59:in `run' > from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout' > from > /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:58:in `run' > from /usr/lib/ruby/site_ruby/1.8/puppet/util/execution.rb:12:in > `withenv' > from > /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:57:in `run' > from > /usr/lib/ruby/site_ruby/1.8/puppet/provider/exec/posix.rb:36:in `chdir' > ... 33 levels... > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:411:in > `exit_on_fail' > from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run' > from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:62:in > `execute' > from /usr/bin/puppet:4 > > Is your catalog maybe installing a new puppet package? That looks like the puppet process is being sent the HUP signal. > Here's what the system is: > > $ cat /etc/*release > CentOS release 5.6 (Final) > $ ruby --version > ruby 1.8.5 (2006-08-25) [x86_64-linux] > $ puppet --version > 2.6.17 > $ rpm -qi puppet > Name : puppet Relocations: (not relocatable) > Version : 2.6.17 Vendor: Fedora Project > Release : 2.el5 Build Date: Thu 19 Jul > 2012 01:25:32 PM EDT > Install Date: Mon 10 Dec 2012 11:37:31 AM EST Build Host: > x86-04.phx2.fedoraproject.org > Group : System Environment/Base Source RPM: > puppet-2.6.17-2.el5.src.rpm > Size : 2534255 License: GPLv2 > Signature : DSA/SHA1, Thu 19 Jul 2012 05:09:16 PM EDT, Key ID > 119cc036217521f6 > Packager : Fedora Project > URL : http://puppetlabs.com > Summary : A network tool for managing many disparate systems > Description : > Puppet lets you centrally manage every important aspect of your system > using a > cross-platform specification language that manages all the separate > elements > normally aggregated in different files, like users, cron jobs, and hosts, > along with obviously discrete elements like packages, services, and files. > > Thanks, > Roman. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
