Issue #1395 has been updated by konrad rzentarzewski.
i'm also getting leaks with storeconfigs and postgresql driver
(puppetmaster=0.25.1; passenger=2.2.2).
now i'm killing threads with >768mb addressed (my server is getting killed
without it).
webrick also was going as high as 2gig (but it was uncapable of handling the
load).
i haven't tried without stored configs. currently 57 nodes connects to this
master.
<pre>
2009-12-08 18:02:01 killed: pid=14966 shared=1500mb (limit: 1024mb)
private=1123mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15061 shared=1480mb (limit: 1024mb)
private=1108mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15169 shared=3046mb (limit: 1024mb)
private=1673mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:05:02 killed: pid=14927 shared=1844mb (limit: 1024mb)
private=1541mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:35:02 killed: pid=20125 shared=3076mb (limit: 1024mb)
private=1594mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:39:01 killed: pid=21464 shared=1389mb (limit: 1024mb)
private=976mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:03:03 killed: pid=24896 shared=1490mb (limit: 1024mb)
private=1078mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:05:01 killed: pid=24795 shared=2634mb (limit: 1024mb)
private=1689mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:06:02 killed: pid=24908 shared=2623mb (limit: 1024mb)
private=1616mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:08:02 killed: pid=24844 shared=1503mb (limit: 1024mb)
private=1254mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:11:02 killed: pid=26456 shared=1391mb (limit: 1024mb)
private=1077mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:34:01 killed: pid=29484 shared=2502mb (limit: 1024mb)
private=1760mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:06:01 killed: pid=1890 shared=1732mb (limit: 1024mb)
private=1179mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:07:02 killed: pid=1691 shared=989mb (limit: 1024mb) private=817mb
(limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:08:01 killed: pid=2834 shared=1470mb (limit: 1024mb)
private=1031mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:11:01 killed: pid=3270 shared=1867mb (limit: 1024mb)
private=1575mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:36:01 killed: pid=6839 shared=1518mb (limit: 1024mb)
private=1101mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:39:01 killed: pid=7218 shared=1534mb (limit: 1024mb)
private=1054mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:41:01 killed: pid=7271 shared=2603mb (limit: 1024mb)
private=1538mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:43:01 killed: pid=7737 shared=1514mb (limit: 1024mb)
private=1266mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:45:02 killed: pid=8394 shared=971mb (limit: 1024mb) private=832mb
(limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:05:02 killed: pid=10905 shared=1729mb (limit: 1024mb)
private=1257mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:12:01 killed: pid=10966 shared=2635mb (limit: 1024mb)
private=1534mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11703 shared=2615mb (limit: 1024mb)
private=1409mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11732 shared=4178mb (limit: 1024mb)
private=1970mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:46:01 killed: pid=12141 shared=1389mb (limit: 1024mb)
private=917mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:08:01 killed: pid=16876 shared=1494mb (limit: 1024mb)
private=1120mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17667 shared=1742mb (limit: 1024mb)
private=1068mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17682 shared=1799mb (limit: 1024mb)
private=1002mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:18:01 killed: pid=18584 shared=2151mb (limit: 1024mb)
private=1318mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:21:01 killed: pid=19036 shared=1659mb (limit: 1024mb)
private=1137mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:40:01 killed: pid=21413 shared=1522mb (limit: 1024mb)
private=1109mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:41:02 killed: pid=21262 shared=1060mb (limit: 1024mb)
private=775mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:48:01 killed: pid=22471 shared=1831mb (limit: 1024mb)
private=1148mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:50:01 killed: pid=23236 shared=1475mb (limit: 1024mb)
private=1042mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:11:01 killed: pid=26442 shared=1777mb (limit: 1024mb)
private=1234mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:20:02 killed: pid=27678 shared=1843mb (limit: 1024mb)
private=1115mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:22:01 killed: pid=28141 shared=2267mb (limit: 1024mb)
private=1471mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:23:02 killed: pid=28457 shared=1174mb (limit: 1024mb)
private=851mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
</pre>
<pre>
/usr/local/bin/ruby /usr/local/bin/passenger-memory-stats | grep "/rack/"|grep
"[0-9] MB"|while read procpid threads shared mb private mb crap ; do
shared=$(echo "$shared"|cut -f1 -d.)
private=$(echo "$private"|cut -f1 -d.)
if [ $shared -gt 1512 ] || [ $private -gt 768 ] ; then
kill -9 $procpid
echo "$(date '+%Y-%m-%d %H:%M:%S') killed: pid=$procpid
shared=${shared}mb (limit: 1024mb) private=${private}mb (limit: 512mb)
process=$crap" >>/tmp/spuchlaki.log
fi
done
</pre>
----------------------------------------
Bug #1395: puppet memory usage
http://projects.reductivelabs.com/issues/1395
Author: Peter Meier
Status: Accepted
Priority: Normal
Assigned to: Markus Roberts
Category: plumbing
Target version: 0.25.2
Patch: None
Affected version: 0.25.1
Keywords: memory leaks
Branch:
In a recent discussion about using puppetd with xinetd ("main
thread":http://groups.google.com/group/puppet-users/browse_thread/thread/50c94008469fc395/11687cd0a633f8f4
) it came out that there is some investigation needed on puppetd's memory
usage.
Different people noticed that running puppetd is using a lot of footprint
(links follow), while other noticed that it uses nearly none (links follow)
Notices about high memory usage:
VIRT/RES
"141m/61m":http://groups.google.com/group/puppet-users/msg/75a8bb3b6652bae6
"127m/82m":http://groups.google.com/group/puppet-users/msg/a4a43d3edd585c03 64
bit
"166m/83m":http://groups.google.com/group/puppet-users/msg/1825ed57ff84287f
"183m/96m":http://groups.google.com/group/puppet-users/msg/ecf2cd36b6e1e3a6
version 0.24.4, 64 bit
Notices about low memory usage:
"nothing
specific":http://groups.google.com/group/puppet-users/msg/aac9fade1b906bd9
version 0.24.4
"57m/46m":http://groups.google.com/group/puppet-users/msg/680b2b2760aa8890
version 0.23.1
Personally I can add another memory usage on a 32bit machine, which is rather
low: 85m virt and 69m res, running 0.24.4
Luke then
"noticed":http://groups.google.com/group/puppet-users/msg/573a76e32fba3eb0 that
we should investigate on memory usage in the near future.
Besides the memory consumption of puppetd, also the memory consumption of
puppetmaster was discussed. There Duncan Hill noticed that splitting the
fileserver off the the main puppetmaster "improved the situation
heavily":http://groups.google.com/group/puppet-users/msg/a7083b931b5c4cda ,
however he
"noticed":http://groups.google.com/group/puppet-users/msg/f68347fdb69b8496 that
the manifest puppetmaster still consumes a lot of memory.
There have been also "other reports about memory usage trashing the master and
the
client":http://groups.google.com/group/puppet-users/browse_thread/thread/26aa8025de704cc4/1c9608f58360cdbb,
however this might not be related.
Quick Summary:
We have seen different environments reporting a lot of memory usage. Most of
them seem to be 64bit machines using 0.24.4. One report about low memory usage
was a 0.23.1 version.
Personal additional assumption: And for a first glance it looks like that 64bit
machines double the memory consumption. (Take care of this assumption)
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://reductivelabs.com/redmine/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
