Issue #4499 has been updated by R.I. Pienaar aka Volcane.
fwiw, if we want to make it faster FasterCSV isnt bad, but its extra requirements. We could perhaps make the code detect the presence of FasterCSV and use that? Two sets of unix timestamps showing parsing my largest CSV file a thousand times: <pre> Before fastercsv: 1282229167 After fastercsv: 1282229169 </pre> <pre> Before csv: 1282229169 After csv: 1282229177 </pre> ---------------------------------------- Refactor #4499: extlookup parses files multiple times http://projects.puppetlabs.com/issues/4499 Author: Alan Barrett Status: Needs design decision Priority: Normal Assigned to: James Turnbull Category: functions Target version: queued Affected version: 0.25.5 Branch: According to "ruby -r profile", more than 50% of the runtime for "puppetmasterd --compile someclient" is in CSV#parse_body. In a typical example, extlookup is called 204 times, CSV::Reader#each is called 225 times (only a little more than the number of calls to extlookup, because almost all variables are found in the first file), CSV#parse_row is called 42486 times (which is not much less than the product of {number of lines in the file} * {number of calls to extlookup}), and CSV#parse_body is called 84747 times (which is approximately double the number of calls to CSV#parse_body). I The code in extlookup.rb appears to read and parse the data files on every call. It would be better if it kept a cache of {file name, variable name, raw value} tuples, invalidated the cache when the file timestamp changed, and otherwise avoided unnecessary re-reading of unchanged files. The decision about which files to search should probably still be done on every call, in case extlookup_datadir or extlookup_precedence has changed. Expansion of "#{variable}" embedded in the raw value should also be done on every call. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
