On Tue, Dec 11, 2012 at 9:21 AM, Roman Shaposhnik <[email protected]>wrote:
> Hi! > > all of a sudden the puppet code that deploys Hadoop > clusters in Bigtop started to trigger: > err: /Stage[main]/Hadoop_head_node/Solr::Server[solrcloud > server]/Solr::Solrcloud_config[collection1]/Exec[ZK collection1 config > upload]: Could not evaluate: Puppet::Util::Log requires a message > > The behavior is not 100% reproducible and it doesn't > happen with the same resource in different runs, but > within the entire run it is almost guaranteed to happen. > So far -- I've seen it triggered for Exec and Package > resources. > Have you been able to come up with a smaller catalog that reproduces the problem? Or something that doesn't include any sensitive information that you could send? > > This happens with quite a few different Puppet versions > I tried (2.6.X, 2.7.X) on various Linux distros (RHEL, SLES) > so I suspect this is a combination of a Puppet issue (perhaps > this one: http://projects.puppetlabs.com/issues/17887 ) > and something in my code that is triggering it. > > I agree with Dominic that this is probably related to the same issue, but I'm curious as to what is causing it. > Taking a hint from the proposed fix to the bug report > I quoted above, I was able to generate the stacktrace > at the point where it happens (attached). > > I would be happy to provide additional details to help > fix this issue in future versions of Puppet, however, > what I'm really, really in need of is any advice > that would help me rejigger the Puppet code so that > I don't bump into this issue that often. > > Looking over the code that is part of the stack trace, I don't see anything obvious. It does look like something has to be happening in the Puppet::Transaction::ResourceHarness#apply_parameter method. Looking over it, the only way we would be getting an empty message is if auditing is not turned on and two exceptions are raised: one needs to be raised in the main section of the method before it has had the chance to set the message and the second has to happen in the rescue before the message gets set there. Those two together would cause the code to execute the `event.log_send` code without ever having received a message. Then when the exception occurs inside the ensure block the original error is lost and the new one is exposed. I'd really like to know what the original error was. My suspicion is that because this is happening and various points it is either an out of memory or related error. Applying Dominic's patch should expose the underlying error that is occurring. > I'm really quite desperate at this point :-( > > Thanks, > Roman. > > Here's the snippet of code I inserted in puppet/util/log.rb > > begin > raise ArgumentError, "Puppet::Util::Log requires a message" unless msg > @message = msg.to_s > rescue => e > @message = "EMPTY MESSAGE: " + e.backtrace.to_s > end > > and here's the resulting stack trace (as you can see it is now failing > in a different resources but failing nonetheless): > > notice: > /Stage[main]/Hadoop_head_node/Hadoop::Create_hdfs_dirs[/var/log]/Exec[HDFS > init /var/log]/returns: EMPTY MESSAGE: > /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:222: > in `message='/usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:203 > in `initialize'/usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:81 > in `new'/usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:81 > in `create'/usr/lib/ruby/site_ruby/1.8/puppet/util/logging.rb:7 > in `send_log'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/event.rb:38 > in > `send_log'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:126 > in > `apply_parameter'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:62 > in > `perform_changes'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:60 > in > `each'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:60 > in > `perform_changes'/usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:133 > in `evaluate'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:48 > in `apply'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:113 > in > `eval_children_and_apply_resource'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:91 > in `eval_resource'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138 > in `evaluate'/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:403 > in `thinmark'/usr/lib/ruby/1.8/benchmark.rb:293 > in `measure'/usr/lib/ruby/1.8/benchmark.rb:307 > in `realtime'/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:402 > in `thinmark'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:137 > in `evaluate'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:130 > in `each'/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:130 > in `evaluate'/usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:150 > in `apply'/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:120 > in > `retrieve_and_apply_catalog'/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:178 > in `benchmark'/usr/lib/ruby/1.8/benchmark.rb:293 > in `measure'/usr/lib/ruby/1.8/benchmark.rb:307 > in `realtime'/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:177 > in `benchmark'/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:119 > in > `retrieve_and_apply_catalog'/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:150 > in `run'/usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:131 > in `main'/usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:35 > in `run_command'/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305 > in `run'/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:420 > in `hook'/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305 > in `run'/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:411 > in `exit_on_fail'/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305 > in `run'/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:62 > in `execute'/usr/bin/puppet:4 > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
