https://bugzilla.wikimedia.org/show_bug.cgi?id=66573
Bug ID: 66573
Summary: Labs: Enable "Puppet freshness" checks in icinga for
cvn project
Product: Wikimedia Labs
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Infrastructure
Assignee: wikibugs-l@lists.wikimedia.org
Reporter: krinklem...@gmail.com
CC: abog...@wikimedia.org, m...@uberbox.org,
rlan...@gmail.com, t...@tim-landscheidt.de
Web browser: ---
Mobile Platform: ---
http://icinga.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=cvn&style=detail
One of my instances was outdated for 2 days because of a stuck lock file that
got left behind when the instance rebooting during a scheduled puppet run. Took
a while to figure out why that one instance wasn't mounting the new /srv.
[07:37 UTC] $ sudo puppet agent -t
Run of Puppet configuration client already in progress; skipping
(/var/lib/puppet/state/agent_catalog_run.lock exists)
[07:37 UTC] $ l /var/lib/puppet/state/agent_catalog_run.lock
-rw-r--r-- 1 root root 0 Jun 12 01:53 agent_catalog_run.lock
syslog
Jun 13 07:23:01 cvn-app5 CRON[27536]: (root) CMD (timeout -k 300 1800 puppet
agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >>
/var/log/puppet.log 2>&1)
Jun 13 07:23:02 cvn-app5 puppet-agent[27538]: Run of Puppet configuration
client already in progress; skipping
(/var/lib/puppet/state/agent_catalog_run.lock exists)
syslog.2.gz
Jun 12 06:23:01 cvn-app5 CRON[5136]: (root) CMD (timeout -k 300 1800 puppet
agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >>
/var/log/puppet.log 2>&1)
Jun 12 06:23:03 cvn-app5 puppet-agent[5138]: Run of Puppet configuration client
already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock
exists)
and going back further:
Jun 12 01:53:01 cvn-app5 CRON[12568]: (root) CMD (timeout -k 300 1800 puppet
agent --onetime --verbose --no-daemonize --splay --splaylimit 60 --show_diff >>
/var/log/puppet.log 2>&1)
Jun 12 01:53:02 cvn-app5 puppet-agent[12570]: Sleeping for 39 seconds (splay is
enabled)
Jun 12 01:53:41 cvn-app5 puppet-agent[12570]: Retrieving plugin
Jun 12 01:53:42 cvn-app5 puppet-agent[12570]: Loading facts in
/var/lib/puppet/lib/facter/puppet_config_dir.rb....
Jun 12 01:55:02 cvn-app5 kernel: imklog 5.8.6, log source = /proc/kmsg started.
Jun 12 01:55:02 cvn-app5 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.6" x-pid="807" x-info="http://www.rsyslog.com";] start
Jun 12 01:55:02 cvn-app5 rsyslogd: rsyslogd's groupid changed to 103
Jun 12 01:55:02 cvn-app5 rsyslogd: rsyslogd's userid changed to 101
Jun 12 01:55:02 cvn-app5 rsyslogd-2039: Could not open output pipe
'/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Initializing cgroup subsys
cpuset
Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Initializing cgroup subsys cpu
Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Linux version 3.2.0-59-virtual
(buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #90-Ubuntu
SMP Tue Jan 7 23:02:33 UTC 2014 (Ubuntu 3.2.0-59.90-virtual 3.2.54)
Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] Command line:
root=UUID=3cbf0c17-65ab-4d22-a2f1-69d1697cf8d0 ro quiet splash console=ttyS0
Jun 12 01:55:02 cvn-app5 kernel: [ 0.000000] KERNEL supported cpus:
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
