Issue #2691 has been updated by Marc Fournier.
> This was because we weren't storing "user" tags in the tag database but as
> parameters of the tagged resources as if they weren't first class citizens
> like system tags. So I made sure the user tags are also stored as tags, and
> taught AR to use those tag tables when querying by tags.
>
> And it seems that AR is doing a poor job, even though I remember testing the
> query plan of the resource.
>From what i've seen, the request itself runs fast (a couple of seconds).
Dealing with the huge amount of data generated seems to be the bottleneck.
As shown below, there are a huge number of (almost) duplicates rows in the
result which let's me think of a missing JOIN in the query.
<pre>
# full dump:
$ wc -l /tmp/data
439345 /tmp/data
# grep on 1 resource:
$ grep "@@check_load on cognac5-mgr" /tmp/data | wc -l
3051
# unique elements composing this resource:
$ grep "@@check_load on cognac5-mgr" /tmp/data | awk '{ print $2 " " $3 " " $4
" " $15 }' | sort | uniq
@@check_load on cognac5-mgr ---
@@check_load on cognac5-mgr admins
@@check_load on cognac5-mgr cognac5-mgr.domain.tld
@@check_load on cognac5-mgr /etc/nagios/auto-puppet/services.cfg
@@check_load on cognac5-mgr generic-service-passive
@@check_load on cognac5-mgr load
@@check_load on cognac5-mgr nagiosserver.domain.tld
@@check_load on cognac5-mgr present
</pre>
The only elements which change between each row are t4_r0 and t4_r1, which
are random integers between 0 and the 439345 (the number of rows returned).
> Can you please use EXPLAIN (or whatever your back-end uses to print the query
> plan) the 0.25 query and post the output here for reference?
> From there, we'll see if we are missing some indices or if we have to write
> the query differently (and then how we can teach AR to issue the correct
> query).
<pre>
id select_type table type possible_keys key key_len ref
rows Extra
1 SIMPLE resources ref
PRIMARY,index_resources_on_host_id,typentitle typentitle 767 const
1512 Using where
1 SIMPLE param_values ref index_param_values_on_resource_id
index_param_values_on_resource_id 5 puppet.resources.id 3
1 SIMPLE param_names eq_ref PRIMARY PRIMARY 4
puppet.param_values.param_name_id 1
1 SIMPLE resource_tags ref
index_resource_tags_on_resource_id,index_resource_tags_on_puppet_tag_id
index_resource_tags_on_resource_id 5 puppet.resources.id 5
Using where
1 SIMPLE puppet_tags eq_ref PRIMARY,index_puppet_tags_on_id PRIMARY
4 puppet.resource_tags.puppet_tag_id 1
Using where
1 SIMPLE resource_tags_puppet_tags ref
index_resource_tags_on_puppet_tag_id index_resource_tags_on_puppet_tag_id
5 puppet.puppet_tags.id 9
</pre>
> One work-around would be to query by a resource parameter meanwhile if it's
> possible for you.
I've setup a 0.24.8 server which uses the same mysql database than the 0.25
server. This works too, of course... Thanks for the idea anyway !
----------------------------------------
Bug #2691: "Could not retrieve catalog: HTTP-Error: 500 Internal Server Error"
with tagged exported resources
http://projects.reductivelabs.com/issues/2691
Author: Marc Fournier
Status: Accepted
Priority: Normal
Assigned to: Brice Figureau
Category:
Target version: 0.25.1
Affected version: 0.25.1rc2
Keywords: rails, storeconfigs
Branch:
The problem happens with this sort of manifest (in fact, this simple example
works fine):
<pre>
node a {
@@file { "/tmp/foo": content => "fjskfjs\n", tag => "foofile", }
}
node b {
File <<| tag == 'foofile' |>>
}
</pre>
In my case, I have a about 20 nodes, each one having:
<pre>
@@nagios_host { $fqdn: tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "a": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "b": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "c": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "d": tag => "nagiosserver.domain.tld", ... }
@@nagios_service { "e": tag => "nagiosserver.domain.tld", ... }
</pre>
I suspect the problem occurs when many exported resources are involved (if 20x5
can be considered many).
The problem happens when running puppet on a node containing:
<pre>
Nagios_host <<| tag == "nagiosserver.domain.tld" |>>
Nagios_service <<| tag == nagiosserver.domain.tld" |>>
</pre>
On the puppetmaster, one ruby process consumes 100% CPU during a few minutes,
then the client says:
<pre>
err: Could not call puppetmaster.getconfig: #<RuntimeError: HTTP-Error: 500
Internal Server Error>
err: Could not retrieve catalog: HTTP-Error: 500 Internal Server Error
</pre>
the ruby process continues running at 100%, and after a while we get this
backtrace:
<pre>
[ pid=3338 file=ext/apache2/Hooks.cpp:547 time=2009-10-01 11:16:46.189 ]:
Backend process 3465 did not return a valid HTTP response. It returned no
data.
*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process
3465):
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in
`write'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:83:in
`process_request'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in
`main_loop'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in
`run'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in
`safe_fork'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in
`spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in
`handle_spawn_application'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`__send__'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`main_loop'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in
`start_synchronously'
from
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61
</pre>
The problem can be produced with 0.24.8 and 0.25.0 clients, and 0.25.0 as well
as 0.25.1rc1 server, with passenger and these gems installed:
activerecord (2.3.2)
activesupport (2.3.2)
fastthread (1.0.7)
passenger (2.2.2)
rack (1.0.0)
rake (0.8.7)
It should be noted that this works fine:
<pre>
Nagios_host <<| |>>
Nagios_service <<| |>>
</pre>
I've also noticed the following error from time to time on 0.25.0. But I'm
unable to reproduce it, and never seen it with 0.25.1rc1, so I'm not sure it's
related:
<pre>
/srv/puppet/lib/puppet/util/settings/file_setting.rb:19:in `group='
/srv/puppet/lib/puppet/util/settings/setting.rb:44:in `send'
/srv/puppet/lib/puppet/util/settings/setting.rb:44:in `initialize'
/srv/puppet/lib/puppet/util/settings/setting.rb:38:in `each'
/srv/puppet/lib/puppet/util/settings/setting.rb:38:in `initialize'
/srv/puppet/lib/puppet/util/settings.rb:398:in `new'
/srv/puppet/lib/puppet/util/settings.rb:398:in `newsetting'
/srv/puppet/lib/puppet/util/settings.rb:533:in `setdefaults'
/srv/puppet/lib/puppet/util/settings.rb:518:in `each'
/srv/puppet/lib/puppet/util/settings.rb:518:in `setdefaults'
/srv/puppet/lib/puppet/reports/store.rb:15:in `mkclientdir'
/srv/puppet/lib/puppet/reports/store.rb:35:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:66:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:59:in `each'
/srv/puppet/lib/puppet/network/handler/report.rb:59:in `process'
/srv/puppet/lib/puppet/network/handler/report.rb:33:in `report'
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `to_proc'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `call'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:52:in `protect_service'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:85:in `setup_processor'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `call'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `each'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:366:in `call_method'
/usr/lib/ruby/1.8/xmlrpc/server.rb:378:in `handle'
/srv/puppet/lib/puppet/network/xmlrpc/processor.rb:44:in `process'
/srv/puppet/lib/puppet/network/http/rack/xmlrpc.rb:35:in `process'
/srv/puppet/lib/puppet/network/http/rack.rb:48:in `call'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/request_handler.rb:81:in
`process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_request_handler.rb:203:in
`main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:110:in
`run'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:67:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/utils.rb:181:in
`safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:60:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/rack/application_spawner.rb:45:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:158:in
`spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/spawn_manager.rb:282:in
`handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:337:in
`main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_passenger/abstract_server.rb:187:in
`start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/bin/passenger-spawn-server:61
</pre>
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://reductivelabs.com/redmine/my/account
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en
-~----------~----~----~----~------~----~------~--~---
