OK, I identified the issue (patch coming soon):
Puppet::SimpleGraph#matching_edge is simply completely inefficient in
front of lots of events for a given resource (which is our case).
The problem is that matching_edge returns the list of edges for which we
should trigger an event. In the aforementioned manifest, since there is
no subscribe/notify, the list is empty.
Unfortunately with the current code, it takes about 22s to find this
result for about 1100 events. With the (soon to be published patch), I
was able to reduce this time to 4s.
Note: 22s doesn't seem that many, but I also tested with about 5000
events and got about 2min (new code), and stopped it after more than
15min (old code).
It seems the previous code was at least quadratic (damn ruby allocator,
not the code itself).
Peter, if you have some spare time, I suggest you try this new patch on
top of the previous one (although they're not related).
Brice, thanks for that follow up. I tested your patch on a directory
containing ~8k of files or directories. Puppet finished chmod and
chown the files as before in about 1-2 minutes. Still it hangs
afterwards burning cpu. However it does not anymore grow an enormous
amount of memory! The amount of memory remains stable now, which is
great.
However after those 1-2 minutes it hangs for about 60 minutes burning
cpu as before and finishes after 62 minutes. However as discussed in
IRC this is imho more a matter of traveling the graph.
To be clear: this is really only a problem within the first run when
puppet touches _every_ file. If it has to touch only a few files it
finishes much faster. For example in about 7 minutes to fix 1k of
files. Which means imho that the main problem remains in passing the
events up.
Still regarding the features and advantages you get by chown/chmoding
such a structure with puppet resources, it would be great if that
could be passed in a reasonable amount of time.
cheers pete
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
