Issue #2691 has been updated by Marc Fournier.
I've tried running the puppetmasterd binary with "--no-daemonize --debug --trace" instead of with passenger, and the problem doesn't occur. But I noticed that much more time is needed to compile the catalog on nodes where tagged exported resources are collected. It takes about 5 minutes to collect my ~100 tagged resources, where normal hosts usually need only 10-15 seconds to compile the catalog. I suppose the error previously described are the consequence of some sort of timeout inside passenger or apache. I have another server with 0.24.8 and passenger, with a node collecting about 300 tagged exported resources, and the catalog gets compiled as fast as on any other node. Maybe this issue can be renamed something like "performance regression with tagged exported resources" ? ---------------------------------------- Bug #2691: "Could not retrieve catalog: HTTP-Error: 500 Internal Server Error" with tagged exported resources http://projects.reductivelabs.com/issues/2691 Author: Marc Fournier Status: Unreviewed Priority: Normal Assigned to: Category: Target version: Affected version: 0.25.1rc1 Keywords: Branch: The problem happens with this sort of manifest (in fact, this simple example works fine): <pre> node a { @@file { "/tmp/foo": content => "fjskfjs\n", tag => "foofile", } } node b { File <<| tag == 'foofile' |>> } </pre> In my case, I have a about 20 nodes, each one having: <pre> @@nagios_host { $fqdn: tag => "nagiosserver.domain.tld", ... } @@nagios_service { "a": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "b": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "c": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "d": tag => "nagiosserver.domain.tld", ... } @@nagios_service { "e": tag => "nagiosserver.domain.tld", ... } </pre> I suspect the problem occurs when many exported resources are involved (if 20x5 can be considered many). The problem happens when running puppet on a node containing: <pre> Nagios_host <<| tag == "nagiosserver.domain.tld" |>> Nagios_service <<| tag == nagiosserver.domain.tld" |>> </pre> On the puppetmaster, one ruby process consumes 100% CPU during a few minutes, then the client says: <pre> err: Could not call puppetmaster.getconfig: #<RuntimeError: HTTP-Error: 500 Internal Server Error> err: Could not retrieve catalog: HTTP-Error: 500 Internal Server Error </pre> the ruby process continues running at 100%, and after a while we get this backtrace: <pre> [ pid=3338 file=ext/apache2/Hooks.cpp:547 time=2009-10-01 11:16:46.189 ]: Backend process 3465 did not return a valid HTTP response. It returned no data. *** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 3465): from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in `write' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in `process_request' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in `main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in `run' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in `safe_fork' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in `spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in `handle_spawn_application' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `__send__' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `main_loop' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in `start_synchronously' from /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 </pre> The problem can be produced with 0.24.8 and 0.25.0 clients, and 0.25.0 as well as 0.25.1rc1 server, with passenger and these gems installed: activerecord (2.3.2) activesupport (2.3.2) fastthread (1.0.7) passenger (2.2.2) rack (1.0.0) rake (0.8.7) It should be noted that this works fine: <pre> Nagios_host <<| |>> Nagios_service <<| |>> </pre> I've also noticed the following error from time to time on 0.25.0. But I'm unable to reproduce it, and never seen it with 0.25.1rc1, so I'm not sure it's related: <pre> /srv/puppet/lib/puppet/util/settings/file_setting.rb:19:in `group=' /srv/puppet/lib/puppet/util/settings/setting.rb:44:in `send' /srv/puppet/lib/puppet/util/settings/setting.rb:44:in `initialize' /srv/puppet/lib/puppet/util/settings/setting.rb:38:in `each' /srv/puppet/lib/puppet/util/settings/setting.rb:38:in `initialize' /srv/puppet/lib/puppet/util/settings.rb:398:in `new' /srv/puppet/lib/puppet/util/settings.rb:398:in `newsetting' /srv/puppet/lib/puppet/util/settings.rb:533:in `setdefaults' /srv/puppet/lib/puppet/util/settings.rb:518:in `each' /srv/puppet/lib/puppet/util/settings.rb:518:in `setdefaults' /srv/puppet/lib/puppet/reports/store.rb:15:in `mkclientdir' /srv/puppet/lib/puppet/reports/store.rb:35:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:66:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:59:in `each' /srv/puppet/lib/puppet/network/handler/report.rb:59:in `process' /srv/puppet/lib/puppet/network/handler/report.rb:33:in `report' /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `to_proc' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `call' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `protect_service' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:85:in `setup_processor' /usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `call' /usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `dispatch' /usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `each' /usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `dispatch' /usr/lib/ruby/1.8/xmlrpc/server.rb:366:in `call_method' /usr/lib/ruby/1.8/xmlrpc/server.rb:378:in `handle' /srv/puppet/lib/puppet/network/xmlrpc/processor.rb:44:in `process' /srv/puppet/lib/puppet/network/http/rack/xmlrpc.rb:35:in `process' /srv/puppet/lib/puppet/network/http/rack.rb:48:in `call' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:81:in `process_request' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in `main_loop' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in `run' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in `safe_fork' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in `spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in `handle_spawn_application' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `__send__' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in `main_loop' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in `start_synchronously' /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61 </pre> -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://reductivelabs.com/redmine/my/account --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en -~----------~----~----~----~------~----~------~--~---
