Issue #4499 has been updated by R.I. Pienaar aka Volcane.
> Let me give my case in more detail. I have more then 200 calls to > extlookup(), most of which are used to fetch version numbers for packages, > which are used in constructs like this: > [...] > There are more than 200 such variables in the CSV files used by extlookup(). > Every time puppetmasterd compiles the catalog, it parses the CSV files more > than 200 times. I have over 200 package version lookups with extlookup as well and my compiles are 2 to 12 seconds. I do not state a version for every package though so my csv file isnt actually 200+ lines big since i tell extlookup to default to 'present' and only set specifics on the packages I care for. This might be masking the problem for me. ---------------------------------------- Refactor #4499: extlookup parses files multiple times http://projects.puppetlabs.com/issues/4499 Author: Alan Barrett Status: Needs design decision Priority: Normal Assigned to: James Turnbull Category: functions Target version: queued Affected version: 0.25.5 Branch: According to "ruby -r profile", more than 50% of the runtime for "puppetmasterd --compile someclient" is in CSV#parse_body. In a typical example, extlookup is called 204 times, CSV::Reader#each is called 225 times (only a little more than the number of calls to extlookup, because almost all variables are found in the first file), CSV#parse_row is called 42486 times (which is not much less than the product of {number of lines in the file} * {number of calls to extlookup}), and CSV#parse_body is called 84747 times (which is approximately double the number of calls to CSV#parse_body). I The code in extlookup.rb appears to read and parse the data files on every call. It would be better if it kept a cache of {file name, variable name, raw value} tuples, invalidated the cache when the file timestamp changed, and otherwise avoided unnecessary re-reading of unchanged files. The decision about which files to search should probably still be done on every call, in case extlookup_datadir or extlookup_precedence has changed. Expansion of "#{variable}" embedded in the raw value should also be done on every call. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
