Issue #2691 has been updated by Brice Figureau.
Marc Fournier wrote: > Christian Hofstaedtler wrote: > > You could - as a possible workaround - try to set Apache's Timeout to > > something larger than 300 seconds and see if that > "helps" (i.e. make the "Broken Pipe" error go away). > > Indeed this works. No more "Broken Pipe" error. I had tried fiddling with > RailsFrameworkSpawnerIdleTime and RailsAppSpawnerIdleTime without success. > Thanks ! > > But this doesn't really solve the problem: puppetmasterd eating a lot of CPU > and inflicting a high load on the server, even with not so many collected > resources. > > > Is it slow too with puppetmasterd instead of Passenger or is it only slow > > with Passenger? > > Yes. I don't think it's a passenger related problem. It's just that I started > investigating in the passenger setup, when getting these "HTTP 500" errors. > Passenger's backtrace didn't really give me a clue about some sort of timeout > occurring. Set the railslog option so that it writes all the requests it sends to the database. Then have a look to the produced requests for the collection (ie not the export part), those should appear first and should be select referencing your tags. Then after that, try to issue manually those requests and EXPLAIN them (or whatever your current database uses to show the query plan). Maybe you're missing an index somewhere. ---------------------------------------- Bug #2691: "Could not retrieve catalog: HTTP-Error: 500 Internal Server Error" with tagged exported resources http://projects.reductivelabs.com/issues/2691 Author: Marc Fournier Status: Unreviewed Priority: Normal Assigned to: Category: Target version: Affected version: 0.25.1rc1 Keywords: Branch: The problem happens with this sort of manifest (in fact, this simple example works fine): <pre> node a { @@file { "/tmp/foo": content => "fjskfjs\n", tag => "foofile", } } node b { File <<| tag == 'foofile' |>> } </pre> In my case, I have a about 20 nodes, each one having: <pre> @@nagios_host { $fqdn: tag => "nagiosserver.domain.tld", ... } @@nagios_service { "a": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "b": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "c": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "d": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "e": tag => "nagiosserver.domain.tld", ... } </pre> I suspect the problem occurs when many exported resources are involved (if 20x5 can be considered many). The problem happens when running puppet on a node containing: <pre> Nagios_host <<| tag == "nagiosserver.domain.tld" |>> Nagios_service <<| tag == nagiosserver.domain.tld" |>> </pre> On the puppetmaster, one ruby process consumes 100% CPU during a few minutes, then the client says: <pre> err: Could not call puppetmaster.getconfig: #<RuntimeError: HTTP-Error: 500 Internal Server Error> err: Could not retrieve catalog: HTTP-Error: 500 Internal Server Error </pre> the ruby process continues running at 100%, and after a while we get this backtrace: <pre> [ pid=3338 file=ext/apache2/Hooks.cpp:547 time=2009-10-01 11:16:46.189 ]: Backend process 3465 did not return a valid HTTP response. It returned no data. *** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 3465): from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in `write' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in `process_request' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in `main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in `run' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in `safe_fork' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in `handle_spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `__send__' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in `start_synchronously' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 </pre> The problem can be produced with 0.24.8 and 0.25.0 clients, and 0.25.0 as well as 0.25.1rc1 server, with passenger and these gems installed: activerecord (2.3.2) activesupport (2.3.2) fastthread (1.0.7) passenger (2.2.2) rack (1.0.0) rake (0.8.7) It should be noted that this works fine: <pre> Nagios_host <<| |>> Nagios_service <<| |>> </pre> I've also noticed the following error from time to time on 0.25.0. But I'm unable to reproduce it, and never seen it with 0.25.1rc1, so I'm not sure it's related: <pre> /srv/puppet/lib/puppet/util/settings/file_setting.rb:19:in `group=' /srv/puppet/lib/puppet/util/settings/setting.rb:44:in `send' /srv/puppet/lib/puppet/util/settings/setting.rb:44:in `initialize' /srv/puppet/lib/puppet/util/settings/setting.rb:38:in `each' /srv/puppet/lib/puppet/util/settings/setting.rb:38:in `initialize' /srv/puppet/lib/puppet/util/settings.rb:398:in `new' /srv/puppet/lib/puppet/util/settings.rb:398:in `newsetting' /srv/puppet/lib/puppet/util/settings.rb:533:in `setdefaults' /srv/puppet/lib/puppet/util/settings.rb:518:in `each' /srv/puppet/lib/puppet/util/settings.rb:518:in `setdefaults' /srv/puppet/lib/puppet/reports/store.rb:15:in `mkclientdir' /srv/puppet/lib/puppet/reports/store.rb:35:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:66:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:59:in `each' /srv/puppet/lib/puppet/network/handler/report.rb:59:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:33:in `report' /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `to_proc' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `call' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `protect_service' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:85:in `setup_processor' /usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `call' /usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `dispatch' /usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `each' /usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `dispatch' /usr/lib/ruby/1.8/xmlrpc/server.rb:366:in `call_method' /usr/lib/ruby/1.8/xmlrpc/server.rb:378:in `handle' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:44:in `process' /srv/puppet/lib/puppet/network/http/rack/xmlrpc.rb:35:in `process' /srv/puppet/lib/puppet/network/http/rack.rb:48:in `call' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:81:in `process_request' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in `main_loop' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in `run' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in `safe_fork' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in `handle_spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `__send__' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `main_loop' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in `start_synchronously' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 </pre> -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
