Issue #19153 has been updated by Charlie Sharpsteen.
Hi Jo,
I have been able to reproduce your issue on Centos 6 using the following
manifest:
service { 'puppet':
ensure => stopped,
enable => false
}
notify { 'Run finished.': }
Service['puppet'] -> Notify['Run finished.']
Using the following crontab:
* * * * * puppet agent --onetime --no-daemon
I get a series of SIGTERM messages in the system log `/var/log/messages` as you
described:
Feb 15 14:39:32 localhost puppet-agent[5182]: Caught TERM; calling stop
Feb 15 14:44:05 localhost puppet-agent[5369]: Caught TERM; calling stop
Feb 15 14:45:03 localhost puppet-agent[5548]: Caught TERM; calling stop
Feb 15 14:46:03 localhost puppet-agent[5748]: Caught TERM; calling stop
Feb 15 14:47:03 localhost puppet-agent[5940]: Caught TERM; calling stop
Feb 15 14:48:03 localhost puppet-agent[6121]: Caught TERM; calling stop
However, if I switch the crontab:
* * * * * puppet agent --onetime --no-daemonize
The kills stop showing up in the log:
Feb 15 14:50:03 localhost puppet-agent[6476]: Run finished.
Feb 15 14:50:03 localhost puppet-agent[6476]: (/Stage[main]//Notify[Run
finished.]/message) defined 'message' as 'Run finished.'
Feb 15 14:50:03 localhost puppet-agent[6476]: Finished catalog run in 0.08
seconds
Feb 15 14:51:03 localhost puppet-agent[6641]: Run finished.
Feb 15 14:51:03 localhost puppet-agent[6641]: (/Stage[main]//Notify[Run
finished.]/message) defined 'message' as 'Run finished.'
Feb 15 14:51:03 localhost puppet-agent[6641]: Finished catalog run in 0.06
seconds
The only other thing I noticed is that if I run a Puppet agent in daemon mode
using `puppet agent --server=foo`, this can also mess things up as the cron-run
agents will refuse to execute if the daemon happens to be holding a lockfile
for catalog updates:
Feb 15 14:52:14 localhost puppet-agent[6967]: Starting Puppet client
version 3.1.0
Feb 15 14:52:14 localhost puppet-agent[6967]: Reopening log files
Feb 15 14:53:02 localhost puppet-agent[6973]: Run of Puppet configuration
client already in progress; skipping
(/var/lib/puppet/state/agent_catalog_run.lock exists)
Feb 15 14:53:23 localhost puppet-agent[6969]: Failed to apply catalog:
Connection timed out - connect(2)
Feb 15 14:54:02 localhost puppet-agent[6980]: Run of Puppet configuration
client already in progress; skipping
(/var/lib/puppet/state/agent_catalog_run.lock exists)
Feb 15 14:54:31 localhost puppet-agent[6969]: Could not send report:
Connection timed out - connect(2)
After switching to `--no-daemonize`, are you still seeing kills of the
cron-controlled agents in the system log or something different?
----------------------------------------
Bug #19153: service puppet ensure stopped kills off cron-run puppet with
"Caught TERM; calling stop"
https://projects.puppetlabs.com/issues/19153#change-83360
Author: Jo Rhett
Status: Investigating
Priority: Urgent
Assignee: Charlie Sharpsteen
Category: agent
Target version: 3.1.1
Affected Puppet version: 3.1.0
Keywords:
Branch:
We have recently switched from puppet agent in daemon mode (for kick) to
cron-run puppet with mcollective agent. However, I started noticing that puppet
policies were being inconsistently applied across the hosts. It turns out that
this policy is the problem:
<pre>
service { 'puppet':
ensure => stopped,
enable => false,
require => File['/etc/cron.d/puppet','/etc/puppet/puppet.conf'],
}
</pre>
I have checked and confirmed that the puppet init script returns the correct
response even when puppet is running. If I run "puppet agent --test" in one
window and while it is running I run this in the other window, it shows clear:
<pre>
root@sj2-noc01 ~$ service puppet status ; echo $?
puppet is stopped
3
</pre>
However, if I run puppet in a silent mode with --onetime and --no-daemon then
the init script returns this value
<pre>
root@sj2-noc01 ~$ service puppet status ; echo $?
puppet (pid 30406) is running...
0
</pre>
This causes it to kill itself off, and not finish the run. Due to the
semi-random nature of ordering, this happens near the end or near the beginning
of the puppet run on different hosts. (there are few dependancies on the puppet
module, so its order in the manifest is random from host to host)
This is clearly a major flaw. We need the above policy to ensure that no puppet
daemons are running, however it interferes with the cron-run instance. However
it is handled properly when run verbosely with test.
Environment is a mixture of CentOS 5 & 6.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
