Hello, I'm running Puppet 2.7.6 and currently expanding the number of servers managed by Puppet. At around the 160-170 host mark (with a 5-minute run interval + splay), my puppetmaster server is starting to die. Is this normal? What I'm getting in the agent logs is the following:
Wed Jun 20 12:15:08 +0200 2012 Puppet (debug): Failed to load library 'selinux' for feature 'selinux' Wed Jun 20 12:15:08 +0200 2012 Puppet (debug): Puppet::Type::File::ProviderWindows: feature microsoft_windows is missing Wed Jun 20 12:15:09 +0200 2012 Puppet (debug): Failed to load library 'ldap' for feature 'ldap' Wed Jun 20 12:15:10 +0200 2012 Puppet (debug): Failed to load library 'shadow' for feature 'libshadow' Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/puppet.conf] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs/ca.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/certs] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/log] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/client_data] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/public_keys/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/public_keys] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/clientbucket] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private_keys/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/private_keys] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/state/graphs] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certificate_requests] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/crl.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/public_keys] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/state/state.yaml] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/client_yaml] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:10 +0200 2012 /File[/var/SP/data/dca/puppet/run] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:10 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private_keys] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/facts] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/state] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/state/last_run_report.yaml] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:11 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/certs] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/state/classes.txt] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/state/last_run_summary.yaml] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/state/resources.txt] (debug): Autorequiring File[/var/SP/data/dca/puppet/state] Wed Jun 20 12:15:11 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config] Wed Jun 20 12:15:11 +0200 2012 /File[/var/SP/data/dca/puppet/lib] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:12 +0200 2012 Puppet (debug): Finishing transaction 2161242520 Wed Jun 20 12:15:12 +0200 2012 Puppet (notice): Reopening log files Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/crl.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/public_keys] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certificate_requests] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs/ca.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/certs] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private_keys] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/certs] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private_keys/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/private_keys] Wed Jun 20 12:15:13 +0200 2012 /File[/var/SP/data/dca/puppet/log] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:13 +0200 2012 /File[/var/SP/data/dca/puppet/state] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/certs] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:13 +0200 2012 /File[/var/SP/data/dca/puppet/lib] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:13 +0200 2012 /File[/var/SP/data/dca/puppet/facts] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:13 +0200 2012 /File[/var/SP/data/dca/puppet/run] (debug): Autorequiring File[/var/SP/data/dca/puppet] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/public_keys/<host-name>.pem] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl/public_keys] Wed Jun 20 12:15:13 +0200 2012 /File[/opt/SP/apps/dca/puppet/config/ssl/private] (debug): Autorequiring File[/opt/SP/apps/dca/puppet/config/ssl] Wed Jun 20 12:15:14 +0200 2012 Puppet (debug): Finishing transaction 2159966420 Wed Jun 20 12:15:14 +0200 2012 Puppet (debug): Using cached certificate for ca Wed Jun 20 12:15:14 +0200 2012 Puppet (debug): Using cached certificate for <host-name> Wed Jun 20 12:15:14 +0200 2012 Puppet (info): Sleeping for 285 seconds (splay is enabled) Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): Finishing transaction 2161291000 Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): Loaded state in 0.05 seconds Wed Jun 20 12:20:01 +0200 2012 Puppet (info): Retrieving plugin Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): Using cached certificate for ca Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): Using cached certificate for <host-name> Wed Jun 20 12:20:01 +0200 2012 Puppet (debug): Using cached certificate_revocation_list for ca Wed Jun 20 12:20:25 +0200 2012 Puppet (debug): Finishing transaction 2160399280 Wed Jun 20 12:20:25 +0200 2012 Puppet (info): Loading facts in env Wed Jun 20 12:20:25 +0200 2012 Puppet (info): Loading facts in opco Wed Jun 20 12:20:25 +0200 2012 Puppet (info): Loading facts in env Wed Jun 20 12:20:25 +0200 2012 Puppet (info): Loading facts in opco Wed Jun 20 12:20:27 +0200 2012 Puppet (debug): catalog supports formats: b64_zlib_yaml dot marshal pson raw yaml; using pson Wed Jun 20 12:20:34 +0200 2012 Puppet (info): Caching catalog for <host-name> Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderDebian: file /usr/sbin/update-rc.d does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderLaunchd: file /bin/launchctl does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderRedhat: file /sbin/service does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderDaemontools: file /usr/bin/svc does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderGentoo: file /sbin/rc-update does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderSystemd: file /bin/systemctl does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Service::ProviderRunit: file /usr/bin/sv does not exist Wed Jun 20 12:20:34 +0200 2012 Puppet (debug): Puppet::Type::Exec::ProviderWindows: feature microsoft_windows is missing Wed Jun 20 12:20:35 +0200 2012 Puppet (debug): Creating default schedules Wed Jun 20 12:20:35 +0200 2012 Puppet (debug): Loaded state in 0.02 seconds Wed Jun 20 12:20:35 +0200 2012 Puppet (info): Applying configuration version '1340179781' Wed Jun 20 12:20:35 +0200 2012 /Stage[main]/mod/Exec[application-reload]/subscribe (debug): subscribes to File[file] Wed Jun 20 12:20:35 +0200 2012 /Stage[main]/mod/File[file] (debug): Autorequiring File[/var/SP/data] Wed Jun 20 12:20:35 +0200 2012 /Schedule[never] (debug): Skipping device resources because running on a host Wed Jun 20 12:20:35 +0200 2012 /Schedule[daily] (debug): Skipping device resources because running on a host Wed Jun 20 12:20:35 +0200 2012 /Schedule[monthly] (debug): Skipping device resources because running on a host Wed Jun 20 12:20:35 +0200 2012 /Schedule[puppet] (debug): Skipping device resources because running on a host Wed Jun 20 12:20:35 +0200 2012 /Schedule[hourly] (debug): Skipping device resources because running on a host Wed Jun 20 12:20:35 +0200 2012 Service[mod](provider=base) (debug): Executing '/usr/ucb/ps -auxww | grep java | grep mod | grep -v grep' Wed Jun 20 12:20:36 +0200 2012 Service[test-service](provider=base) (debug): Executing '/usr/ucb/ps -auxww | grep test-service | grep -v grep' Wed Jun 20 12:20:37 +0200 2012 Puppet (debug): file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson Wed Jun 20 12:20:38 +0200 2012 Puppet (debug): file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson Wed Jun 20 12:22:40 +0200 2012 Puppet (debug): Storing state Wed Jun 20 12:22:40 +0200 2012 Puppet (debug): Stored state in 0.15 seconds Wed Jun 20 12:22:41 +0200 2012 Puppet (debug): Value of 'preferred_serialization_format' (pson) is invalid for report, using default (b64_zlib_yaml) Wed Jun 20 12:22:41 +0200 2012 Puppet (debug): report supports formats: b64_zlib_yaml marshal raw yaml; using b64_zlib_yaml Wed Jun 20 12:23:18 +0200 2012 Puppet (err): Could not run Puppet configuration client: execution expired I've already gone through a few threads mentioning this same error, but it's still not clear to me where the failure is happening exactly (I'm thinking as the agent tries to send the report to the puppetmaster?) or whether anything can be done about it. The puppetmasterd process seems to be using 90-100% of one CPU, but there is another one. Are there any parameters I can tweak on the puppetmaster server to help it handle the load better? Regards, Kmbu -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/85EzWtt3qKkJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
