Hi, Apologies for the delay in replying. I completely missed the posts!
I am yet to find a solution. I ended up reverting to 2.7.23 until I had more time to investigate the issue. Laurent, have you had any success in the past week or so? Thanks for the heads up James. I'm not explicitly setting vardir, but will certainly do so. The puppet master is running passenger on RHEL 6. I will try 3.3.2, however if I can't be assured the problem won't occur again when I roll it into production, I am hesitant. Because of the randomness it is really hard to replicate, and didn't show up in my test environment. Perhaps it relates to load on the puppet master??? Lou On Fri, Nov 29, 2013 at 10:39 PM, <[email protected]> wrote: > Hi, > > For reference, I've just upgraded my puppet masters from 2.7.22 to 3.3.2 > and haven't seen any errors of this kind. > > I presume you are running with passenger? I am too. CentOS EL6 masters. > > Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this for > you both. > > I have seen one nasty other error though. > > If you aren't setting the vardir explicitly in all clients puppet.conf, > I'd suggest doing so before upgrading. > After my upgrade, I had to manually intervene on almost all client nodes > because puppet was failing to run. Bug filed at > http://projects.puppetlabs.com/issues/23311 > > J > > On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >> >> I am running into the exact same issue with 3.3.1 Did you find a solution >> for it? >> >> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>> >>> Hello, >>> >>> I have a rhel 6 puppet master with the following packages installed: >>> >>> facter.x86_64 1:1.7.3-1.el6 >>> hiera.noarch 1.2.1-1.el6 >>> puppet.noarch 3.3.1-1.el6 >>> puppet-server.noarch 3.3.1-1.el6 >>> ruby.x86_64 1.8.7.352-12.el6_4 >>> >>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>> downloaded from the puppetlabs yum repo. >>> >>> I am now randomly seeing the following errors: >>> >>> 1. On the node I get: >>> … >>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using >>> pson >>> Error: Could not retrieve catalog from remote server: Error 500 on >>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>> <html><head> >>> <title>500 Internal Server Error</title> >>> </head><body> >>> <h1>Internal Server Error</h1> >>> <p>The server encountered an internal error or >>> misconfiguration and was unable to complete >>> your request.</p> >>> <p>Please contact the server administrator, >>> root@localhost and inform them of the time the error occurred, >>> and anything you might have done that may have >>> caused the error.</p> >>> <p>More information about this error may be available >>> in the server error log.</p> >>> <hr> >>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> >>> </body></html> >>> >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>> `is_http_200?' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>> `find' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>> `retrieve_new_catalog' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark' >>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>> `retrieve_new_catalog' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>> `retrieve_catalog' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>> `prepare_and_retrieve_catalog' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>> `controlled_run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>> `onetime' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>> `run_command' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>> `plugin_hook' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>> `execute' >>> /opt/csw/bin/puppet:4 >>> Warning: Not using cache on failed catalog >>> Error: Could not retrieve catalog; skipping run >>> >>> 2. In /var/log/messages I see errors such as: >>> >>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>> <node> in environment production in 33.30 seconds >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >>> is deprecated and will be removed in a future version. See >>> http://links.puppetlabs.com/deprecate_yaml_on_network >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>> `response_formatter_for') >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >>> is deprecated and will be removed in a future version. See >>> http://links.puppetlabs.com/deprecate_yaml_on_network >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>> `request_format') >>> >>> 3. And in the apache error log I get errors such as: >>> >>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of >>> script headers: <node1> >>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >>> not send a valid HTTP response; instead, it sent nothing at all. It is >>> possible that it has crashed; please check whether there are crashing bugs >>> in this application. >>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>> >>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of >>> script headers: <node2> >>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >>> not send a valid HTTP response; instead, it sent nothing at all. It is >>> possible that it has crashed; please check whether there are crashing bugs >>> in this application. >>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>> Segmentation fault >>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>> >>> --- >>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris >>> 10 servers to 3.2.4 (as that is the latest csw package available). >>> >>> The vast majority of the errors occur with the solaris servers running >>> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >>> Annoyingly I have had the error on a rhel server once, so I can’t say for >>> certain the Solaris servers are the problem :/ >>> >>> I have searched for similar errors but cannot find anything exact. Any >>> help would be greatly appreciated. >>> >>> Many thanks, >>> Lou >>> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
