Thanks for the response ... will do. Just to clarify though, were you getting the internal server errors every time the agent ran for a particular node, or were they sometimes successful?
Lou On Wed, Dec 11, 2013 at 8:47 AM, Laurent Domb <[email protected]>wrote: > Hi Louise, > > In my case the error was triggered by a faulty manifest which contained a > variable ::$fqdn instead of $::fqdn. So it might be worth looking at the > manifests and check for faulty vars. > > Laurent > On Dec 10, 2013 5:29 PM, "Louise Baker" <[email protected]> wrote: > >> >> Hi, >> >> Apologies for the delay in replying. I completely missed the posts! >> >> I am yet to find a solution. I ended up reverting to 2.7.23 until I had >> more time to investigate the issue. Laurent, have you had any success in >> the past week or so? >> >> Thanks for the heads up James. I'm not explicitly setting vardir, but >> will certainly do so. The puppet master is running passenger on RHEL6. I >> will try 3.3.2, however if I can't be assured the problem won't occur >> again when I roll it into production, I am hesitant. Because of the >> randomness it is really hard to replicate, and didn't show up in my test >> environment. Perhaps it relates to load on the puppet master??? >> >> Lou >> >> On Fri, Nov 29, 2013 at 10:39 PM, <[email protected]> wrote: >> >>> Hi, >>> >>> For reference, I've just upgraded my puppet masters from 2.7.22 to 3.3.2 >>> and haven't seen any errors of this kind. >>> >>> I presume you are running with passenger? I am too. CentOS EL6 masters. >>> >>> Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this >>> for you both. >>> >>> I have seen one nasty other error though. >>> >>> If you aren't setting the vardir explicitly in all clients puppet.conf, >>> I'd suggest doing so before upgrading. >>> After my upgrade, I had to manually intervene on almost all client nodes >>> because puppet was failing to run. Bug filed at >>> http://projects.puppetlabs.com/issues/23311 >>> >>> J >>> >>> On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >>>> >>>> I am running into the exact same issue with 3.3.1 Did you find a >>>> solution for it? >>>> >>>> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have a rhel 6 puppet master with the following packages installed: >>>>> >>>>> facter.x86_64 1:1.7.3-1.el6 >>>>> hiera.noarch 1.2.1-1.el6 >>>>> puppet.noarch 3.3.1-1.el6 >>>>> puppet-server.noarch 3.3.1-1.el6 >>>>> ruby.x86_64 1.8.7.352-12.el6_4 >>>>> >>>>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>>>> downloaded from the puppetlabs yum repo. >>>>> >>>>> I am now randomly seeing the following errors: >>>>> >>>>> 1. On the node I get: >>>>> … >>>>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; >>>>> using pson >>>>> Error: Could not retrieve catalog from remote server: Error 500 on >>>>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>>> <html><head> >>>>> <title>500 Internal Server Error</title> >>>>> </head><body> >>>>> <h1>Internal Server Error</h1> >>>>> <p>The server encountered an internal error or >>>>> misconfiguration and was unable to complete >>>>> your request.</p> >>>>> <p>Please contact the server administrator, >>>>> root@localhost and inform them of the time the error occurred, >>>>> and anything you might have done that may have >>>>> caused the error.</p> >>>>> <p>More information about this error may be available >>>>> in the server error log.</p> >>>>> <hr> >>>>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port >>>>> 8140</address> >>>>> </body></html> >>>>> >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>>>> `is_http_200?' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in >>>>> `find' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>>>> `find' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>>>> `retrieve_new_catalog' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark' >>>>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>>>> `retrieve_new_catalog' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>>>> `retrieve_catalog' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>>>> `prepare_and_retrieve_catalog' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>>>> `controlled_run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>>>> `onetime' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>>>> `run_command' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>>>> `plugin_hook' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in >>>>> `run' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>>>> `execute' >>>>> /opt/csw/bin/puppet:4 >>>>> Warning: Not using cache on failed catalog >>>>> Error: Could not retrieve catalog; skipping run >>>>> >>>>> 2. In /var/log/messages I see errors such as: >>>>> >>>>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>>>> <node> in environment production in 33.30 seconds >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>> requests is deprecated and will be removed in a future version. See >>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>>>> `response_formatter_for') >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>> requests is deprecated and will be removed in a future version. See >>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>>>> `request_format') >>>>> >>>>> 3. And in the apache error log I get errors such as: >>>>> >>>>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end >>>>> of script headers: <node1> >>>>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) >>>>> did >>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>> possible that it has crashed; please check whether there are crashing bugs >>>>> in this application. >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>> >>>>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end >>>>> of script headers: <node2> >>>>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) >>>>> did >>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>> possible that it has crashed; please check whether there are crashing bugs >>>>> in this application. >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>> Segmentation fault >>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>> >>>>> --- >>>>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>>>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some >>>>> solaris >>>>> 10 servers to 3.2.4 (as that is the latest csw package available). >>>>> >>>>> The vast majority of the errors occur with the solaris servers running >>>>> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >>>>> Annoyingly I have had the error on a rhel server once, so I can’t say for >>>>> certain the Solaris servers are the problem :/ >>>>> >>>>> I have searched for similar errors but cannot find anything exact. >>>>> Any help would be greatly appreciated. >>>>> >>>>> Many thanks, >>>>> Lou >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com >>> . >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZdBt_46NYvY5ViJAzq57PNS1JFzoEseozgqaCm_%3DG_Cg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
