Issue #7938 has been updated by Daniel Pittman.
https://github.com/daniel-pittman/puppet-dashboard/commits/feature/master/7938-background-tasks-for-dashboard has the current working state of this. It includes the gem import, clean versions of the code using it, and integrates it everywhere. It should be passing tests correctly and everything. Needs final polish, and some administrative / documentation consideration before it is finally polished and shipped. ---------------------------------------- Feature #7938: Accept 4K reports in one hour to dashboard https://projects.puppetlabs.com/issues/7938 Author: Daniel Pittman Status: Accepted Priority: Normal Assignee: Daniel Pittman Category: Target version: Keywords: Branch: https://github.com/daniel-pittman/puppet-dashboard/commits/bug/master/7689-parallel-import-of-reports Affected URL: Affected Dashboard version: To meet the performance goals we have set, we need to be able to deliver either 2K or 4K reports to dashboard in a single hour. That is potentially more than one report per second, although we can reasonably spend up to 16 cores on this problem. (Less would be nice.) Given the performance of ActiveRecord, it doesn't seem achievable that we can hit that during the current round of work. We only need the reports *delivered* during that window, however. So, if we accept at line-rate, buffer behind, and import the reports into the server more slowly we should be able to get the performance required. The most promising model for this is to build a report "spooler" that sits beside the regular report import tool. This can accept the report data, then spool it for later processing in a buffer that can accept the data. We then import that data as fast as possible – we have ~ 2 hours for 2K nodes – while still being able to deliver the accept performance required. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-bugs?hl=en.
