From: Pieter van de Bruggen <[email protected]> Delayed_Job is a gem that implements robust background processing for Ruby tasks. Now we have vendored it, we can start to use it to process report import in the background.
This has two advantages: one, we can import in parallel, since we can have more than one worker in the background working on the YAML transformation and database updating. Two, we now return to the report submitter in short order: all we have to do is spool the YAML to disk, and register the background job, and we are assured that we will eventually get that into the database. This eliminates a lot of the master => dashboard submission pipeline delay, so that the master gets back to serving clients immediately, rather than having to wait until the report is entirely ingested. Reviewed-By: Daniel Pittman <[email protected]> Reviewed-By: Matt Robinson <[email protected]> --- README.markdown | 16 +++++++++- app/models/report.rb | 4 ++ config/environment.rb | 1 + config/initializers/delayed_job.rb | 24 +++++++++++++++ db/migrate/20110614234202_create_delayed_jobs.rb | 31 ++++++++++++++++++++ .../templates/script => script/delayed_job | 0 6 files changed, 74 insertions(+), 2 deletions(-) create mode 100644 config/initializers/delayed_job.rb create mode 100644 db/migrate/20110614234202_create_delayed_jobs.rb copy vendor/plugins/delayed_job/generators/delayed_job/templates/script => script/delayed_job (100%) diff --git a/README.markdown b/README.markdown index 2c18322..599bec7 100644 --- a/README.markdown +++ b/README.markdown @@ -105,7 +105,12 @@ Installation 2. Create a `config/database.yml` file to specify Puppet Dashboard's database configuration. Please see the `config/database.yml.example` file for further details about database configurations and environments. These files paths are relative to the path of the Puppet Dashboard software containing this `README.markdown` file. -3. Setup a MySQL database server, create a user and database for use with the Puppet Dashboard by either: +3. Configure MySQL maximum packet size, to permit larger rows in the database. Puppet Dashboard can send up to 17MB of data in a single row, although it is extraordinarily rare that it will. You should configure your server `my.cnf` to increase the limit to at least 24MB (32MB or more recommended), and restart MySQL for this to take effect. (See [the MySQL documentation][http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_max_allowed_packet] for more details, and how to up the limit without restarting.) + + # Allowing 32MB ensures our 17MB row with plenty of spare room + max_allowed_packet = 32M + +4. Setup a MySQL database server, create a user and database for use with the Puppet Dashboard by either: 1. Using a `rake` task to create just the database from settings in the `config/database.yml` file. You must `cd` into the directory with the Puppet Dashboard software containing this `README.markdown` file before running these commands: @@ -117,7 +122,7 @@ Installation CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'my_password'; GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost'; -4. Populate the database with the tables for the Puppet Dashboard. +5. Populate the database with the tables for the Puppet Dashboard. 1. For typical use with the `production` environment: @@ -343,6 +348,13 @@ Third-party tools that can help secure a Puppet Dashboard include: 4. HTTPS (SSL) Encryption is supported when running Dashboard under Apache and Passenger. The example configuration in `ext/passenger/dashboard-vhost.conf` includes a commented-out vhost configured to use SSL. You may need to change the Apache directives SSLCertificateFile, SSLCertificateKeyFile, SSLCACertificateFile, and SSLCARevocationFile to the paths of the files created by the `cert` rake tasks. (See `Generating certs and connecting to the puppet master` for how to create these files) +Background Processing +--------------------- + +The Puppet Dashboard performs a number of tasks, such as report import, that can consume significant system resources. To ensure that performance remains snappy under load, we use the `delayed_job` background processing system to manage these tasks without tying up a web front end thread. + +The Puppet Dashboard code will automatically spawn a manager and worker in the background, which will run these tasks without tying up the resources of the web server. This will share the same credentials and access as the web front-end, so should introduce no additional security risk. + Performance ----------- diff --git a/app/models/report.rb b/app/models/report.rb index 1a49e2b..3fc20e3 100644 --- a/app/models/report.rb +++ b/app/models/report.rb @@ -96,6 +96,10 @@ class Report < ActiveRecord::Base report end + class << self + handle_asynchronously :create_from_yaml + end + def assign_to_node self.node = Node.find_or_create_by_name(self.host) end diff --git a/config/environment.rb b/config/environment.rb index 308ff19..73d54b8 100644 --- a/config/environment.rb +++ b/config/environment.rb @@ -17,6 +17,7 @@ Rails::Initializer.run do |config| config.gem 'sass' config.gem 'will_paginate' config.gem 'maruku' + config.gem 'daemons', :version => '1.0.10' # Change this to adjust log rotation. Logger.new(log_file, number_of_logs, max_log_size). config.logger = Logger.new("#{RAILS_ROOT}/log/#{RAILS_ENV}.log", 50, 10.megabytes) diff --git a/config/initializers/delayed_job.rb b/config/initializers/delayed_job.rb new file mode 100644 index 0000000..29183bd --- /dev/null +++ b/config/initializers/delayed_job.rb @@ -0,0 +1,24 @@ +DELAYED_JOB_PID_PATH = "#{Rails.root}/tmp/pids/delayed_job.pid" + +Delayed::Worker.destroy_failed_jobs = false +Delayed::Worker.max_attempts = 3 + +def start_delayed_job + Thread.new do + `#{Rails.root}/script/delayed_job -p dashboard -m start` + end +end + +def process_is_dead? + begin + pid = File.read(DELAYED_JOB_PID_PATH).strip + Process.kill(0, pid.to_i) + false + rescue + true + end +end + +if !File.exist?(DELAYED_JOB_PID_PATH) && process_is_dead? + start_delayed_job +end diff --git a/db/migrate/20110614234202_create_delayed_jobs.rb b/db/migrate/20110614234202_create_delayed_jobs.rb new file mode 100644 index 0000000..ecb7db9 --- /dev/null +++ b/db/migrate/20110614234202_create_delayed_jobs.rb @@ -0,0 +1,31 @@ +class CreateDelayedJobs < ActiveRecord::Migration + def self.up + create_table :delayed_jobs, :force => true do |table| + # Allows some jobs to jump to the front of the queue + table.integer :priority, :default => 0 + # Provides for retries, but still fail eventually. + table.integer :attempts, :default => 0 + # YAML-encoded string of the object that will do work + table.text :handler, :limit => 16.megabytes + # reason for last failure (See Note below) + table.text :last_error + # When to run. Could be Time.zone.now for immediately, or sometime in + # the future. + table.datetime :run_at + # Set when a client is working on this object + table.datetime :locked_at + # Set when all retries have failed (actually, by default, the record is + # deleted instead) + table.datetime :failed_at + # Who is working on this object (if locked) + table.string :locked_by + table.timestamps + end + + add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority' + end + + def self.down + drop_table :delayed_jobs + end +end diff --git a/vendor/plugins/delayed_job/generators/delayed_job/templates/script b/script/delayed_job similarity index 100% copy from vendor/plugins/delayed_job/generators/delayed_job/templates/script copy to script/delayed_job -- 1.7.5.4 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
