Issue #2691 has been updated by Marc Fournier.
Brice, thanks for the patch !
It does work better, but I'm not sure it's quite as efficient as with 0.24.8
though. The query now generates a lot less rows, but still way too many.
In my case, 153 collected resources return 79675 rows. Although it doesn't
completely bring the puppetmaster down anymore, the load still peaks during
a couple of minutes and I fear this will not scale for people with more
collected resources.
Here is the SQL query that gets generated:
<pre>
SELECT `resources`.`id` AS t0_r0, `resources`.`title` AS t0_r1,
`resources`.`restype` AS t0_r2, `resources`.`host_id` AS t0_r3,
`resources`.`source_file_id` AS t0_r4, `resources`.`exported` AS t0_r5,
`resources`.`line` AS t0_r6, `resources`.`updated_at` AS t0_r7,
`resources`.`created_at` AS t0_r8, `puppet_tags`.`id` AS t1_r0,
`puppet_tags`.`name` AS t1_r1, `puppet_tags`.`updated_at` AS t1_r2,
`puppet_tags`.`created_at` AS t1_r3, `resource_tags_puppet_tags`.`id` AS t2_r0,
`resource_tags_puppet_tags`.`resource_id` AS t2_r1,
`resource_tags_puppet_tags`.`puppet_tag_id` AS t2_r2,
`resource_tags_puppet_tags`.`updated_at` AS t2_r3,
`resource_tags_puppet_tags`.`created_at` AS t2_r4 FROM `resources` LEFT OUTER
JOIN `resource_tags` ON (`resources`.`id` = `resource_tags`.`resource_id`) LEFT
OUTER JOIN `puppet_tags` ON (`puppet_tags`.`id` =
`resource_tags`.`puppet_tag_id`) LEFT OUTER JOIN `resource_tags`
resource_tags_puppet_tags ON resource_tags_puppet_tags.puppet_tag_id =
puppet_tags.id WHERE (host_id != 6 AND (exported=1 AND
restype='Nagios_service') AND (puppet_tags.name =
'nagios-nagiosserver.domain.tld'))
</pre>
Here is the query plan:
<pre>
+----+-------------+---------------------------+--------+-------------------------------------------------------------------------+--------------------------------------+---------+------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys
| key
| key_len | ref | rows | Extra |
+----+-------------+---------------------------+--------+-------------------------------------------------------------------------+--------------------------------------+---------+------------------------------------+------+-------------+
| 1 | SIMPLE | resources | ref |
PRIMARY,index_resources_on_host_id,typentitle |
typentitle | 767 | const
| 1208 | Using where |
| 1 | SIMPLE | resource_tags | ref |
index_resource_tags_on_resource_id,index_resource_tags_on_puppet_tag_id |
index_resource_tags_on_resource_id | 5 | puppet.resources.id
| 5 | Using where |
| 1 | SIMPLE | puppet_tags | eq_ref |
PRIMARY,index_puppet_tags_on_id |
PRIMARY | 4 |
puppet.resource_tags.puppet_tag_id | 1 | Using where |
| 1 | SIMPLE | resource_tags_puppet_tags | ref |
index_resource_tags_on_puppet_tag_id |
index_resource_tags_on_puppet_tag_id | 5 | puppet.puppet_tags.id
| 9 | |
+----+-------------+---------------------------+--------+-------------------------------------------------------------------------+--------------------------------------+---------+------------------------------------+------+-------------+
</pre>
And here is some sample output:
<pre>
t0_r0 t0_r1 t0_r2 t0_r3 t0_r4 t0_r5 t0_r6 t0_r7 t0_r8 t1_r0
t1_r1 t1_r2 t1_r3 t2_r0 t2_r1 t2_r2 t2_r3 t2_r4
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
14544 1
306 86 2009-10-15 18:06:31 2009-10-15 18:06:31
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
15563 1
397 86 2009-10-15 18:06:38 2009-10-15 18:06:38
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
15850 1
423 86 2009-10-15 18:06:40 2009-10-15 18:06:40
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
17148 1
540 86 2009-10-15 18:06:49 2009-10-15 18:06:49
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
17953 1
613 86 2009-10-15 18:06:54 2009-10-15 18:06:54
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
20615 1
852 86 2009-10-15 18:07:18 2009-10-15 18:07:18
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
21653 1
943 86 2009-10-15 18:07:30 2009-10-15 18:07:30
38793 @@check_load on cognac5-mgr Nagios_service 41 575 1
39 2009-10-21 01:31:26 2009-10-20 01:31:30 86
nagios-nagiosserver.domain.tld 2009-10-15 18:00:43 2009-10-15 18:00:43
29115 2
643 86 2009-10-15 18:22:20 2009-10-15 18:22:20
[...]
</pre>
----------------------------------------
Bug #2691: "Could not retrieve catalog: HTTP-Error: 500 Internal Server Error"
with tagged exported resources
http://projects.reductivelabs.com/issues/2691
Author: Marc Fournier
Status: Ready for Testing
Priority: Normal
Assigned to: Brice Figureau
Category:
Target version: 0.25.1
Affected version: 0.25.1rc2
Keywords: rails, storeconfigs
Branch:
The problem happens with this sort of manifest (in fact, this simple example
works fine):
<pre>
node a {
@@file { "/tmp/foo": content => "fjskfjs\n", tag => "foofile", }
}
node b {
File <<| tag == 'foofile' |>>
}
</pre>
In my case, I have a about 20 nodes, each one having:
<pre>
@@nagios_host { $fqdn: tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "a": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "b": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "c": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "d": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "e": tag => "nagiosserver.domain.tld", ... }
</pre>
I suspect the problem occurs when many exported resources are involved (if 20x5
can be considered many).
The problem happens when running puppet on a node containing:
<pre>
Nagios_host <<| tag == "nagiosserver.domain.tld" |>>
Nagios_service <<| tag == nagiosserver.domain.tld" |>>
</pre>
On the puppetmaster, one ruby process consumes 100% CPU during a few minutes,
then the client says:
<pre>
err: Could not call puppetmaster.getconfig: #<RuntimeError: HTTP-Error: 500
Internal Server Error>
err: Could not retrieve catalog: HTTP-Error: 500 Internal Server Error
</pre>
the ruby process continues running at 100%, and after a while we get this
backtrace:
<pre>
[ pid=3338 file=ext/apache2/Hooks.cpp:547 time=2009-10-01 11:16:46.189 ]:
Backend process 3465 did not return a valid HTTP response. It returned no
data.
*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process
3465):
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in
`write'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in
`process_request'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in
`main_loop'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in
`run'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in
`safe_fork'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in
`handle_spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`__send__'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`main_loop'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in
`start_synchronously'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61
</pre>
The problem can be produced with 0.24.8 and 0.25.0 clients, and 0.25.0 as well
as 0.25.1rc1 server, with passenger and these gems installed:
activerecord (2.3.2)
activesupport (2.3.2)
fastthread (1.0.7)
passenger (2.2.2)
rack (1.0.0)
rake (0.8.7)
It should be noted that this works fine:
<pre>
Nagios_host <<| |>>
Nagios_service <<| |>>
</pre>
I've also noticed the following error from time to time on 0.25.0. But I'm
unable to reproduce it, and never seen it with 0.25.1rc1, so I'm not sure it's
related:
<pre>
/srv/puppet/lib/puppet/util/settings/file_setting.rb:19:in `group='
/srv/puppet/lib/puppet/util/settings/setting.rb:44:in `send'
/srv/puppet/lib/puppet/util/settings/setting.rb:44:in `initialize'
/srv/puppet/lib/puppet/util/settings/setting.rb:38:in `each'
/srv/puppet/lib/puppet/util/settings/setting.rb:38:in `initialize'
/srv/puppet/lib/puppet/util/settings.rb:398:in `new'
/srv/puppet/lib/puppet/util/settings.rb:398:in `newsetting'
/srv/puppet/lib/puppet/util/settings.rb:533:in `setdefaults'
/srv/puppet/lib/puppet/util/settings.rb:518:in `each'
/srv/puppet/lib/puppet/util/settings.rb:518:in `setdefaults'
/srv/puppet/lib/puppet/reports/store.rb:15:in `mkclientdir'
/srv/puppet/lib/puppet/reports/store.rb:35:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:66:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:59:in `each'
/srv/puppet/lib/puppet/network/handler/report.rb:59:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:33:in `report'
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `to_proc'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `call'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `protect_service'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:85:in `setup_processor'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `call'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `each'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:366:in `call_method'
/usr/lib/ruby/1.8/xmlrpc/server.rb:378:in `handle'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:44:in `process'
/srv/puppet/lib/puppet/network/http/rack/xmlrpc.rb:35:in `process'
/srv/puppet/lib/puppet/network/http/rack.rb:48:in `call'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:81:in
`process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in
`main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in
`run'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in
`safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in
`handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in
`start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61
</pre>
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://reductivelabs.com/redmine/my/account
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en
-~----------~----~----~----~------~----~------~--~---
