Issue #11360 has been reported by Jo Rhett.
----------------------------------------
Bug #11360: puppet client hangs after period of being unable to contact server
https://projects.puppetlabs.com/issues/11360
Author: Jo Rhett
Status: Unreviewed
Priority: Normal
Assignee:
Category: agent
Target version:
Affected Puppet version: 2.6.12
Keywords:
Branch:
We had some serious memory/swap issues with the puppet master today. I spent a
few hours getting that worked out, and upgrading to passenger 3.0.11. After
clearing up the issues we found that some 50 systems weren't up to date.
Logging into these systems I found a puppetdlock file which was about 4.5 hours
old and a running puppetd which was looping doing nothing.
[03:30 root@ald002 ~]$ ls -la /var/lib/puppet/state
total 164
drwxr-xr-t 3 root root 4096 Dec 12 22:45 .
drwxr-xr-x 10 puppet puppet 4096 Oct 24 18:28 ..
drwxr-xr-x 2 root root 4096 Sep 23 05:03 graphs
-rw-rw---- 1 root root 1448 Dec 12 22:15 last_run_report.yaml
-rw-rw---- 1 root root 38 Dec 12 22:15 last_run_summary.yaml
-rw-r--r-- 1 root root 4 Dec 12 22:45 puppetdlock
-rw-rw---- 1 root root 109285 Dec 12 22:15 state.yaml
Here's an example log:
Dec 12 21:12:27 ald002 puppet-agent[6945]: Could not retrieve catalog from
remote server: Connection refused - connect(2)
Dec 12 21:12:27 ald002 puppet-agent[6945]: Using cached catalog
Dec 12 21:12:27 ald002 puppet-agent[6945]: Could not retrieve catalog; skipping
run
Dec 12 21:12:27 ald002 puppet-agent[6945]: Could not send report: Connection
refused - connect(2)
Dec 12 21:42:30 ald002 puppet-agent[6945]: Could not retrieve catalog from
remote server: Connection refused - connect(2)
Dec 12 21:42:30 ald002 puppet-agent[6945]: Using cached catalog
Dec 12 21:42:30 ald002 puppet-agent[6945]: Could not retrieve catalog; skipping
run
Dec 12 21:42:30 ald002 puppet-agent[6945]: Could not send report: Connection
refused - connect(2)
Dec 12 22:15:57 ald002 puppet-agent[6945]: Could not run Puppet configuration
client: execution expired
[03:32 root@ald002 ~]$ strace -p 6945
Process 6945 attached - interrupt to quit
select(11, [7 9], [], [], {1, 107000}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 0}) = 0 (Timeout)
select(11, [9], [], [], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
select(11, [7 9], [], [], {1, 999999}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 0}) = 0 (Timeout)
select(11, [9], [], [], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
select(11, [7 9], [], [], {1, 999999}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 602}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 0}) = 0 (Timeout)
select(11, [9], [], [], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
select(11, [7 9], [], [], {1, 999997}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 566}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 0}) = 0 (Timeout)
select(11, [9], [], [], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
select(11, [7 9], [], [], {1, 999998}) = 0 (Timeout)
select(11, [7 9], [], [], {0, 0}) = 0 (Timeout)
select(11, [9], [], [], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
select(11, [7 9], [], [], {1, 999999} <unfinished ...>
Process 6945 detached
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
