On 8/7/14 5:41 AM, Chris Ritson wrote:
I've been trying for a while to work out why my puppet agent run on
the puppet master was consuming so much time and memory. With a
single webrick master running puppet puppet-3.6.2-1.el6 and serving
about 100 clients, I was seeing agent runs on the master of the
order of 4000 seconds. I was keeping, but not making use of the agent
reports, so the puppet master was tidying these away after they
reached the an age of 36 hours to avoid wasting disk space. Agent
runs were quite often getting OOM errors and being killed as this was
on a virtual machine with 2G of memory. Increasing this limit and
watching the agent run grow it became clear that stored state was
part of the problem. Memory consumption was sometimes reaching 3G.
In the end, even after abandoning report generation and allowing the
remaining report files to be tidied out of existence, I was still
seeing a large memory footprint and long delays when the puppet run
appeared to be doing nothing or saving its previous state. Looking in
the saved state.yaml file, I saw that there were still many hundreds
of references to report files and directories that used to exist.
Only when I finally removed this file altogether, taking the risk
that a few services would be restarted when they shouldn't be, did my
puppet run get back to a reasonable time and memory footprint.
It seems from this that the puppet agent is reading and preserving a
lot of historic state information, even when this is no longer of any
use. Is this a bug/feature? Is this something I could/should have
been able to avoid with better knowledge of puppet's configuration
options?
Chris Ritson
You're not alone with this problem.
http://ask.puppetlabs.com/question/13092/puppetmaster-process-memory-usage-constantly-increases-until-kernel-complains-and-starts-killing-off-other-processes/
I do think that there is something not quite right about whatever Puppet
is doing in more recent versions. That said moving your master to the
Apache/Passenger setup should solve it and is the correct solution
regardless of the memory problems you're having. webrick masters were
never meant to be the means to run Puppet in production or at more than
10 nodes in my opinion.
Ramin
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/53E66FCF.8090102%40badapple.net.
For more options, visit https://groups.google.com/d/optout.
