No the error occurred randomly. Once I found the module which seemed to cause the issue and excluded all other module from the node declaration I could reproduce the error on every single puppetrun.
On Tue, Dec 10, 2013 at 6:11 PM, Louise Baker <[email protected]>wrote: > Thanks for the response ... will do. Just to clarify though, were you > getting the internal server errors every time the agent ran for a > particular node, or were they sometimes successful? > > Lou > > > On Wed, Dec 11, 2013 at 8:47 AM, Laurent Domb <[email protected]>wrote: > >> Hi Louise, >> >> In my case the error was triggered by a faulty manifest which contained a >> variable ::$fqdn instead of $::fqdn. So it might be worth looking at the >> manifests and check for faulty vars. >> >> Laurent >> On Dec 10, 2013 5:29 PM, "Louise Baker" <[email protected]> wrote: >> >>> >>> Hi, >>> >>> Apologies for the delay in replying. I completely missed the posts! >>> >>> I am yet to find a solution. I ended up reverting to 2.7.23 until I had >>> more time to investigate the issue. Laurent, have you had any success in >>> the past week or so? >>> >>> Thanks for the heads up James. I'm not explicitly setting vardir, but >>> will certainly do so. The puppet master is running passenger on RHEL6. I >>> will try 3.3.2, however if I can't be assured the problem won't occur >>> again when I roll it into production, I am hesitant. Because of the >>> randomness it is really hard to replicate, and didn't show up in my test >>> environment. Perhaps it relates to load on the puppet master??? >>> >>> Lou >>> >>> On Fri, Nov 29, 2013 at 10:39 PM, <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> For reference, I've just upgraded my puppet masters from 2.7.22 to >>>> 3.3.2 and haven't seen any errors of this kind. >>>> >>>> I presume you are running with passenger? I am too. CentOS EL6 >>>> masters. >>>> >>>> Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this >>>> for you both. >>>> >>>> I have seen one nasty other error though. >>>> >>>> If you aren't setting the vardir explicitly in all clients puppet.conf, >>>> I'd suggest doing so before upgrading. >>>> After my upgrade, I had to manually intervene on almost all client >>>> nodes because puppet was failing to run. Bug filed at >>>> http://projects.puppetlabs.com/issues/23311 >>>> >>>> J >>>> >>>> On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >>>>> >>>>> I am running into the exact same issue with 3.3.1 Did you find a >>>>> solution for it? >>>>> >>>>> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have a rhel 6 puppet master with the following packages installed: >>>>>> >>>>>> facter.x86_64 1:1.7.3-1.el6 >>>>>> hiera.noarch 1.2.1-1.el6 >>>>>> puppet.noarch 3.3.1-1.el6 >>>>>> puppet-server.noarch 3.3.1-1.el6 >>>>>> ruby.x86_64 1.8.7.352-12.el6_4 >>>>>> >>>>>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>>>>> downloaded from the puppetlabs yum repo. >>>>>> >>>>>> I am now randomly seeing the following errors: >>>>>> >>>>>> 1. On the node I get: >>>>>> … >>>>>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; >>>>>> using pson >>>>>> Error: Could not retrieve catalog from remote server: Error 500 on >>>>>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>>>> <html><head> >>>>>> <title>500 Internal Server Error</title> >>>>>> </head><body> >>>>>> <h1>Internal Server Error</h1> >>>>>> <p>The server encountered an internal error or >>>>>> misconfiguration and was unable to complete >>>>>> your request.</p> >>>>>> <p>Please contact the server administrator, >>>>>> root@localhost and inform them of the time the error occurred, >>>>>> and anything you might have done that may have >>>>>> caused the error.</p> >>>>>> <p>More information about this error may be available >>>>>> in the server error log.</p> >>>>>> <hr> >>>>>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port >>>>>> 8140</address> >>>>>> </body></html> >>>>>> >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>>>>> `is_http_200?' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in >>>>>> `find' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>>>>> `find' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>>>>> `retrieve_new_catalog' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark' >>>>>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>>>>> `retrieve_new_catalog' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>>>>> `retrieve_catalog' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>>>>> `prepare_and_retrieve_catalog' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>>>>> `controlled_run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>>>>> `onetime' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>>>>> `run_command' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>>>>> `plugin_hook' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in >>>>>> `run' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>>>>> `execute' >>>>>> /opt/csw/bin/puppet:4 >>>>>> Warning: Not using cache on failed catalog >>>>>> Error: Could not retrieve catalog; skipping run >>>>>> >>>>>> 2. In /var/log/messages I see errors such as: >>>>>> >>>>>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>>>>> <node> in environment production in 33.30 seconds >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>>> requests is deprecated and will be removed in a future version. See >>>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>>>>> `response_formatter_for') >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>>> requests is deprecated and will be removed in a future version. See >>>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>>>>> `request_format') >>>>>> >>>>>> 3. And in the apache error log I get errors such as: >>>>>> >>>>>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end >>>>>> of script headers: <node1> >>>>>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) >>>>>> did >>>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>>> possible that it has crashed; please check whether there are crashing >>>>>> bugs >>>>>> in this application. >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>>> >>>>>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end >>>>>> of script headers: <node2> >>>>>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) >>>>>> did >>>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>>> possible that it has crashed; please check whether there are crashing >>>>>> bugs >>>>>> in this application. >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>>> Segmentation fault >>>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>>> >>>>>> --- >>>>>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>>>>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some >>>>>> solaris >>>>>> 10 servers to 3.2.4 (as that is the latest csw package available). >>>>>> >>>>>> The vast majority of the errors occur with the solaris servers >>>>>> running 2.7.19, but I have had a 3.2.4 fail consistently with the same >>>>>> error. Annoyingly I have had the error on a rhel server once, so I can’t >>>>>> say for certain the Solaris servers are the problem :/ >>>>>> >>>>>> I have searched for similar errors but cannot find anything exact. >>>>>> Any help would be greatly appreciated. >>>>>> >>>>>> Many thanks, >>>>>> Lou >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com >>>> . >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Puppet Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com >>> . >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAPWqdEZdBt_46NYvY5ViJAzq57PNS1JFzoEseozgqaCm_%3DG_Cg%40mail.gmail.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOcP4mz4LNy6FWkkv7tOJjuPPa0HvohFN91idngRF2ZMUQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
