Issue #1973 has been updated by masterzen.
Preamble:
You didn't provide any information about how long puppetd is using 100% of CPU,
is it always or only when it checks out its configuration from the master.
Can you also try to run one of your client in debug mode, then look at the log
when puppetd is using 100% of cpu to see what it is doing.
That'll help diagnose the issue.
harisekhon wrote:
> ruby-1.8.5 and ruby-1.8.6
>
> I have used one or two recursive file copies, but their depth is usually no
> more than 1/2 dirs.
How many files are there (on the client side, not on the source side)?
> - Shouldn't puppet only do md5 sums for files for which remote copies are
> found?
That's the whole point of bug #1469.
> - Shouldn't the remote copies already have an MD5 from a puppetmaster where
> puppet:// is used instead of having puppetd repeat this MD5 across the whole
> infrastructure needlessly?
The source MD5 is computed by the master I guess, but each client has to
compute the local version of the checksum file. Someone could have changed the
file behind puppet's back.
> If I was to write a program in python to md5 10-20 files, I would expect that
> to finish in 1-2 secs, not be taking minutes of 100% cpu and causing OOM on
> loaded machines.
I never said it was the cause of your issues. I just wanted to point that if
you have on your clients tons of files that are behing a recursive file{} with
source/checksum on, it can take some time and memory for puppet to stat(1) all
this files, builds the checksums (and cache it).
If you have only a handful of file, then this is certainly not the issue.
> (I do not want to disable md5 checking in favour of ctime/mtime)
>
> - Does puppet still use md5s if you are only setting perms and not sourcing
> files? (ie no source param is given)
My understanding is that any file {} resource with checksum => XXX will compute
the checksum, independently of using a source or not.
> - I want recurse true and not 2/3 when ensuring perms on a directory tree
> consisting of 50-100 files, it should only be statting them and be really
> fast, right?
Yes, provided it doesn't compute checksum on those. Maybe try "checksum =>
undef" on those recursive file{} can help?
I tried the following little example:
<pre>
File { checksum => md5 }
file {
"/tmp/test":
recurse => true, mode => 0644
}
</pre>
Under /tmp/test I have several dirs and files.
let's look at the result
<pre>
[br...@arsenic]% RUBYLIB=`pwd`/lib bin/puppet --debug --trace /tmp/test.pp
debug: File[/tmp/test]: Adding default for checksum
debug: Creating default schedules
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Failed to load library 'shadow' for feature 'libshadow'
debug: Finishing transaction 69992060055820 with 0 changes
debug: //File[/tmp/test]/checksum: Initializing checksum hash
debug: //File[/tmp/test]/checksum: computed checksum
debug: //File[/tmp/test]: Creating checksum {mtime}Thu Feb 19 11:11:49 +0100
2009
debug: //File[/tmp/test/a]/checksum: Initializing checksum hash
debug: //File[/tmp/test/a]/checksum: computed checksum
debug: //File[/tmp/test/a]: Creating checksum {mtime}Thu Feb 19 11:11:50 +0100
2009
debug: //File[/tmp/test/a/b]/checksum: Initializing checksum hash
debug: //File[/tmp/test/a/b]/checksum: computed checksum
debug: //File[/tmp/test/a/b]: Creating checksum {mtime}Thu Feb 19 11:11:51
+0100 2009
debug: //File[/tmp/test/a/b/c]/checksum: Initializing checksum hash
debug: //File[/tmp/test/a/b/c]/checksum: computed checksum
debug: //File[/tmp/test/a/b/c]: Creating checksum {mtime}Thu Feb 19 11:11:58
+0100 2009
debug: //File[/tmp/test/a/b/c/e]/checksum: Initializing checksum hash
debug: //File[/tmp/test/a/b/c/e]/checksum: computed checksum
debug: //File[/tmp/test/a/b/c/e]: Creating checksum
{md5}d41d8cd98f00b204e9800998ecf8427e
debug: Finishing transaction 69992062707440 with 0 changes
</pre>
Note how puppet created checksums, even though you don't need them.
Using this manifest:
<pre>
File { checksum => md5 }
file {
"/tmp/test":
recurse => true, mode => 0644n checksum => undef
}
</pre>
I get the correct result:
<pre>
[br...@arsenic]% RUBYLIB=`pwd`/lib bin/puppet --debug --trace /tmp/test.pp
debug: Creating default schedules
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Failed to load library 'shadow' for feature 'libshadow'
debug: Finishing transaction 69843146739140 with 0 changes
debug: Finishing transaction 69843149395900 with 0 changes
</pre>
Note: this time it didn't compute unneeded checksums.
> I've been through every recurse in every manifest (and I have a default of
> false in site)
>
> There are only a few recurses set for dirs in which there are no source files
> set, only owner/group/mode settings, so if these are MD5ing without
> intelligence, then this would explain a lot.
So my point: if you have checksum => md5 (or mtime) as default, and you don't
want checksum computations on every file for those recursive file {},
selectively checksum => undef on them and see if that's better.
If you want to really debug the issue, disable puppetd on one of your server,
then launch puppetd manually with --test --debug --trace and see in the console
output if there are checksum computations for lots of files.
----------------------------------------
Bug #1973: Performance problems
http://projects.reductivelabs.com:80/issues/1973
Author: harisekhon
Status: Needs more information
Priority: High
Assigned to:
Category:
Target version:
Complexity: Unknown
Affected version: 0.24.6
Keywords:
Puppetd is chewing up a ridiculous amount of CPU and the boss at work has
unfortunately noticed and described it as "unacceptable".
I would really like to see puppet run more efficiently. We are running fast
multi-cpu servers with multi-cores have servers running oracle databases and
computational services so when someone says that puppet is making a noticeable
dent in cpu time, I don't have much of a come back for that, other than "that's
life!". I understand that the multi-cpus and core probably makes no difference
as I believe this is pretty much single threaded but the point is they are new
and fast servers.
My fear is that my boss will turn around and say "Get rid of that bloatware, I
want my servers to go fast..."
Unfortunately my boss is not the only person who has called puppet a resource
hog... help!
----------------------------------------
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://reductivelabs.com/redmine/my/account
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en
-~----------~----~----~----~------~----~------~--~---
