You guys are the best, quite the quick turn around! I'll be applying this to our largest puppetmaster tomorrow and running a series of tests so I can give you some feedback.
On Thu, May 10, 2012 at 2:44 PM, Nick Lewis <[email protected]> wrote: > On Thursday, May 10, 2012 at 10:30 AM, Ashley Penney wrote: > > On Thu, May 10, 2012 at 1:42 AM, Jeff Weiss <[email protected]>wrote: > > > Fourth, Peter, Émile, and Trevor (or anyone else experiencing the > problem), would you be willing to be pre-release testers of improvements? > Our ops team is seeing the problem too, but that's only a single real-world > data point. We need to make sure we don't self-optimize. We need your > help to make sure the performance fixes address *your* problems not just > ours. > > > I just wanted to chime in to offer to test any performance patches because > I'm definitely interested in helping get to the bottom of any performance > drops. We also see the massive memory use during run issues other people > have seen which has resulted in us stopping puppet runs on certain > over-commited boxes as it can tip them over the edge. Anything that > targets either of this areas is something I'll happily hack in by hand for > testing purposes. (I have 4 puppetmasters currently so it's easy to target > just a few and not the rest for comparisons). > > > On that note: After spending a few hours hunting yesterday, we identified > the cause of a significant performance regression in compilation when using > modules with metadata. I've attached a patch for anyone who would like to > test it out. We're seeing approximately 5-10x faster compilation. > > The basic gist of the issue is that every time Puppet looks up a class > (via include/import/autoload), it constructs a list of possible modules the > class could be in (based on its name). For each module, it will create a > Puppet::Module instance, so that it can ask which manifests in that module > may contain the class. Unfortunately, creating the Puppet::Module instance > will cause the metadata.json file to be read and parsed. Given that this > happens several times *per include*, it gets expensive quickly. > > Fortunately, as we already have a list of Puppet::Module objects for all > the modules, the simple solution is to just use one of those. > > Once again, a patch is attached. Please test this and provide feedback. > Note that this patch should not be applied to Puppet 2.6, or to 2.7.3 or > earlier. Due to a bug in caching, it will cause significantly *worse* > performance. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
