Issue #11381 has been updated by Jo Rhett.
Well here's another spike and look at this: that's like 33 rack processes.
Wtf? Ignore the config much?
<pre>
[20:48 root@us0401ats003 puppet]$ ps auwx |grep puppet
puppet 2789 3.0 0.6 214624 107808 ? R 20:11 1:11 Rack:
/nas/puppet/rack
puppet 2793 3.2 0.6 213052 106104 ? R 20:11 1:18 Rack:
/nas/puppet/rack
puppet 3036 2.5 0.6 212976 105840 ? R 20:13 0:57 Rack:
/nas/puppet/rack
puppet 6733 0.2 0.2 141088 42180 ? S 20:45 0:00 master
puppet 6879 7.8 0.6 210152 103256 ? S 20:46 0:24 Rack:
/nas/puppet/rack
puppet 7135 6.9 0.4 177000 76336 ? S 20:48 0:13 Rack:
/nas/puppet/rack
puppet 7141 8.1 0.4 177780 76848 ? R 20:48 0:14 Rack:
/nas/puppet/rack
puppet 7146 7.1 0.4 177416 76596 ? R 20:48 0:12 Rack:
/nas/puppet/rack
puppet 7163 5.8 0.4 177696 77036 ? R 20:48 0:09 Rack:
/nas/puppet/rack
puppet 7356 0.3 0.0 0 0 ? Z 20:50 0:00 [ruby]
<defunct>
puppet 7419 0.0 0.4 177072 75048 ? S 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7420 0.0 0.0 58388 3668 ? R 20:51 0:00
/usr/sbin/sendmail [email protected]
puppet 7440 0.5 0.6 222784 114680 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7443 0.6 0.6 219012 110912 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7447 0.2 0.8 250580 142176 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7449 0.8 0.0 0 0 ? R 20:51 0:00 [ruby]
puppet 7457 1.0 0.6 213912 105932 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7461 1.5 0.0 0 0 ? Z 20:51 0:00 [ruby]
<defunct>
puppet 7465 1.1 0.8 250544 142260 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7470 1.5 0.6 211836 103540 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7472 1.8 0.6 218732 110612 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7474 0.8 0.6 219436 111316 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7475 1.5 0.6 218084 110112 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7477 0.2 0.6 212048 103600 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7478 2.2 0.8 244936 136940 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7479 3.0 0.6 218208 110084 ? R 20:51 0:00 Rack:
/nas/puppet/rack
root 7485 0.0 0.0 61200 744 pts/1 R+ 20:51 0:00 grep puppet
puppet 7486 4.0 0.6 217404 109284 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7487 1.5 0.6 222820 114596 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7488 3.5 0.0 0 0 ? Z 20:51 0:00 [ruby]
<defunct>
puppet 7489 13.0 0.0 0 0 ? Z 20:51 0:00 [ruby]
<defunct>
puppet 7492 9.0 0.8 244944 136940 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7493 0.0 0.4 177368 75832 ? R 20:51 0:00 Rack:
/nas/puppet/rack
puppet 7494 0.0 0.0 0 0 ? Z 20:51 0:00 [ruby]
<defunct>
root 10614 1.5 0.9 266452 163584 ? Ssl 02:19 16:50 /usr/bin/ruby
/usr/sbin/puppetd --server=puppetmaster.tangome.gbl --logdest=syslog --listen
puppet 29417 3.2 0.6 220756 113872 ? R 19:13 3:08 Rack:
/nas/puppet/rack
puppet 29757 2.6 0.6 218800 111976 ? R 19:21 2:22 Rack:
/nas/puppet/rack
puppet 30610 4.0 0.7 223508 116632 ? R 19:38 2:56 Rack:
/nas/puppet/rack
puppet 30656 3.5 0.6 218120 111172 ? R 19:38 2:35 Rack:
/nas/puppet/rack
puppet 30663 4.9 0.6 219448 112832 ? R 19:38 3:37 Rack:
/nas/puppet/rack
puppet 30669 2.8 0.8 245628 138816 ? S 19:38 2:04 Rack:
/nas/puppet/rack
puppet 30692 2.5 0.6 218544 111724 ? R 19:38 1:53 Rack:
/nas/puppet/rack
puppet 30759 2.5 0.6 213132 106068 ? R 19:38 1:53 Rack:
/nas/puppet/rack
puppet 30833 3.1 0.8 250064 143244 ? R 19:38 2:19 Rack:
/nas/puppet/rack
puppet 30871 3.0 0.6 218900 111964 ? S 19:38 2:12 Rack:
/nas/puppet/rack
puppet 30908 3.6 0.8 251256 144476 ? R 19:38 2:38 Rack:
/nas/puppet/rack
puppet 32015 1.8 0.6 213060 106148 ? R 19:41 1:15 Rack:
/nas/puppet/rack
</pre>
----------------------------------------
Bug #11381: puppetmaster death spiral under passenger -- document the needs!
https://projects.puppetlabs.com/issues/11381
Author: Jo Rhett
Status: Needs More Information
Priority: Normal
Assignee: Jo Rhett
Category: passenger
Target version:
Affected Puppet version: 2.6.12
Keywords:
Branch:
Having run a cfengine master server that handled 25k clients, I guess I should
feel spoiled. But the apparent system requirements for puppetmaster are
phenomenal. With a mere 500 nodes we have a dedicated machine with 4 cores, 8
GB of memory and 6GB of swap, and yet puppetmaster goes into a death spiral
daily. There is nothing on this host other than apache, passenger and
puppetmaster. (and nrpe/nagios test to ensure puppet client is running)
This is what top looks like when it happens:
<pre>
top - 01:18:06 up 1 day, 1:53, 2 users, load average: 185.70, 148.74, 77.73
Tasks: 379 total, 181 running, 198 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 99.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.1%hi, 0.1%si, 0.0%st
Mem: 8174508k total, 8132764k used, 41744k free, 524k buffers
Swap: 6094840k total, 6094840k used, 0k free, 19784k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7938 puppet 18 0 216m 100m 648 R 43.0 1.3 0:02.65 ruby
31786 puppet 19 0 215m 107m 1724 R 34.1 1.3 2:46.71 ruby
364 root 15 0 0 0 0 S 13.2 0.0 1:21.89 pdflush
7868 puppet 19 0 217m 102m 648 R 11.4 1.3 0:05.21 ruby
8028 root 15 0 0 0 0 S 11.4 0.0 0:21.73 pdflush
7804 puppet 19 0 212m 96m 648 R 11.1 1.2 0:02.38 ruby
7802 puppet 18 0 243m 131m 840 R 7.4 1.6 0:06.40 ruby
7692 puppet 19 0 212m 16m 648 R 7.1 0.2 0:06.10 ruby
7573 puppet 18 0 210m 12m 648 R 6.1 0.2 0:13.12 ruby
7900 puppet 18 0 225m 111m 648 R 6.1 1.4 0:05.88 ruby
7926 puppet 19 0 215m 105m 648 R 6.1 1.3 0:03.42 ruby
7941 puppet 18 0 181m 79m 648 R 6.1 1.0 0:02.68 ruby
7561 puppet 18 0 200m 21m 648 R 5.8 0.3 0:13.21 ruby
7792 puppet 18 0 222m 113m 940 R 4.9 1.4 0:11.08 ruby
8113 root 19 0 102m 896 608 R 4.9 0.0 0:01.40 crond
7902 puppet 18 0 209m 100m 852 R 4.3 1.3 0:04.42 ruby
7429 puppet 18 0 207m 25m 648 R 4.0 0.3 0:10.24 ruby
31816 puppet 19 0 225m 117m 1652 R 4.0 1.5 2:28.63 ruby
7685 puppet 18 0 210m 19m 648 R 3.7 0.2 0:10.95 ruby
7918 puppet 18 0 215m 101m 648 R 3.7 1.3 0:03.52 ruby
8121 root 18 0 60476 1144 800 R 3.4 0.0 0:00.73 sshd
31825 puppet 18 0 220m 110m 1652 R 3.4 1.4 2:54.23 ruby
7417 puppet 19 0 198m 30m 648 R 3.1 0.4 0:10.72 ruby
7459 puppet 19 0 206m 17m 648 R 3.1 0.2 0:08.91 ruby
7479 puppet 19 0 199m 17m 648 R 3.1 0.2 0:09.01 ruby
7570 puppet 18 0 205m 19m 648 R 3.1 0.2 0:14.22 ruby
7576 puppet 19 0 212m 12m 648 R 3.1 0.2 0:08.61 ruby
7585 puppet 19 0 207m 18m 648 R 3.1 0.2 0:07.44 ruby
7589 puppet 19 0 204m 14m 648 R 3.1 0.2 0:07.00 ruby
7593 puppet 19 0 181m 81m 1548 R 3.1 1.0 0:37.07 ruby
7620 puppet 19 0 210m 17m 648 R 3.1 0.2 0:07.81 ruby
7625 puppet 19 0 209m 21m 648 R 3.1 0.3 0:08.22 ruby
7652 puppet 18 0 164m 10m 648 R 3.1 0.1 0:03.61 ruby
7656 puppet 19 0 213m 35m 648 R 3.1 0.5 0:18.16 ruby
7669 puppet 19 0 204m 23m 648 R 3.1 0.3 0:10.32 ruby
7672 puppet 19 0 207m 14m 648 R 3.1 0.2 0:06.61 ruby
7676 puppet 20 0 205m 17m 648 R 3.1 0.2 0:07.71 ruby
7708 puppet 18 0 208m 16m 648 R 3.1 0.2 0:04.46 ruby
7739 puppet 19 0 221m 14m 648 R 3.1 0.2 0:04.93 ruby
7743 puppet 19 0 212m 34m 648 R 3.1 0.4 0:04.51 ruby
7747 puppet 19 0 207m 25m 648 R 3.1 0.3 0:08.15 ruby
7794 puppet 19 0 213m 41m 648 R 3.1 0.5 0:07.06 ruby
7842 puppet 18 0 211m 100m 648 R 3.1 1.3 0:06.48 ruby
7850 puppet 19 0 212m 96m 852 R 3.1 1.2 0:05.51 ruby
7852 puppet 19 0 212m 95m 648 R 3.1 1.2 0:01.68 ruby
7855 puppet 19 0 209m 97m 924 R 3.1 1.2 0:10.06 ruby
7872 puppet 19 0 214m 97m 852 R 3.1 1.2 0:08.38 ruby
</pre>
1. Passenger clients are limited to 20. Where did all these other ruby
instances come from? (there is no other ruby code on the system)
2. Why is it willing to spawn until system death? How can I limit this?
CentOS 5.7 with ruby 1.8.5 and all puppet packages from yum.puppetlabs.com
Passenger 3.0.11 at the moment but we first saw this with passenger 2.2 and
upgraded without any change in behavior.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
