Hi, Since the below is a little long, I put my question at the top: how do I troubleshoot awful exported resources performance in puppet and is there anything I can tweak to get it to run under 10 minutes in larger environments?
I have a fairly modest environment (118 nodes, but prod will be at least twice as large). I'm trying to move my distributed nagios setup to one based on exported resources and it works brilliantly in a smaller env (35 nodes), but with 118 nodes it's not so great anymore. All the hosts export a nagios host and services (up to 20) with corresponding commands, plus files for them. It's all based on this: https://github.com/camptocamp/puppet-nagios , but fixed so that it's working :). I appreciate it's not the most efficient way of doing things, but it makes my setup insanely easy to configure, comparing to the previous one, with a combination of static files, templates, multiple 'profiles', duplicated configs, etc. In the small env puppet agent takes about 2 minutes to run on nagios server and collect resources. In the larger env it takes about 70 minutes, if it manages to finish at all. Initially, as a "quick" test, I was running puppetdb without postgres and had to give it 5GB to get it to finish at all (70 mins). With postgres 8.4, load on the puppetmaster is significantly reduced, but with 512MB for puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb still runs out of memory. I set it to 1GB and puppedb just crashed again (I've got dumps). Trying with 2GB now. I haven't fiddled with thread settings, but my puppet agents aren't deamonized or 'croned', I run them using mcollective or manually. So there's only a single puppet agent running during this test, on the core nagios server. It seems that there's a ruby process taking 100% of one core during this run and nothing else "dramatic" seems to be happening (except for puppetdb dying of course). It's all on centos 5.8 with puppetlabs ruby and puppet packages, on apache+passenger: ruby-1.8.7.370-1.el5 ruby-rdoc-1.8.7.370-1.el5 rubygem-stomp-1.2.2-1.el5 rubygem-json-1.4.6-2.el5 rubygem-fastthread-1.0.7-1 rubygem-rake-0.8.7-2 rubygem-daemon_controller-0.2.5-1 rubygem-passenger-native-3.0.12-1 puppetdb-terminus-1.0.5-1.el5 puppet-server-3.0.2-1.el5 ruby-libs-1.8.7.370-1.el5 ruby-irb-1.8.7.370-1.el5 rubygems-1.3.7-1.el5 ruby-shadow-1.4.1-7 libselinux-ruby-1.33.4-5.7.el5 ruby-augeas-0.4.1-1 puppet-3.0.2-1.el5 rubygem-rack-1.1.0-2.el5 rubygem-passenger-3.0.12-1 rubygem-passenger-native-libs-3.0.12-1_1.8.7.370 puppetdb-1.0.5-1.el5 Passenger config: PassengerHighPerformance on PassengerMaxPoolSize 12 PassengerPoolIdleTime 1500 PassengerMaxRequests 1000 PassengerStatThrottleRate 120 Regards, Daniel -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/bLnpN2PTnQsJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
