On Friday, May 11, 2012 at 1:25 PM, Peter Meier wrote: > > On that note: After spending a few hours hunting yesterday, we > > identified the cause of a significant performance regression in > > compilation when using modules with metadata. I've attached a patch > > for anyone who would like to test it out. We're seeing approximately > > 5-10x faster compilation. > > > > > Awesome work! > > > The basic gist of the issue is that every time Puppet looks up a > > class (via include/import/autoload), it constructs a list of possible > > modules the class could be in (based on its name). For each module, > > it will create a Puppet::Module instance, so that it can ask which > > manifests in that module may contain the class. Unfortunately, > > creating the Puppet::Module instance will cause the metadata.json > > file to be read and parsed. Given that this happens several times > > *per include*, it gets expensive quickly. > > > > Fortunately, as we already have a list of Puppet::Module objects for > > all the modules, the simple solution is to just use one of those. > > > > Once again, a patch is attached. Please test this and provide > > feedback. Note that this patch should not be applied to Puppet 2.6, > > or to 2.7.3 or earlier. Due to a bug in caching, it will cause > > significantly *worse* performance. > > > > > Question: I have only like 2-5 Modules (of >120) that have a > metadata.json file. Would this make sense to test it as well in my > situation? I didn't yet understand whether even looking for it is > already an expensive process as well or mainly reading the metadata.json > file is expensive. > But looking at your patch it sounds anyway better to not create a lot of > unnecessary objects. > > So maybe this would also explain my compilation time increase I can see > with 2.7. > >
Well it's probably worth trying at least, since the patch is simple. I think the difference really depends on which particular modules have metadata, and how your modules are related. If you often refer to nested classes from within those modules, or refer to nested classes that live in those modules, it may definitely help. Also, even if the files don't exist, it's still a lot of stats, which could add up. This would definitely be an interesting data point. > > Thanks > > ~pete > > PS: I assume I fine that patch also in your git repo? -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
