On Wed, Jul 25, 2012 at 03:21:25AM -0700, Chad wrote:
>
> We are building a service on ruote, and have encountered some performance
> problems. We originally used Route::Sequel::Storage with the mysql2
> adapter. On deployment we noticed that ruote made massive number of sql
> requests, producing numerous time outs.
Hello Chad,
do you have details? I'd like to make ruote-sequel better.
> For comparison, we switched to Route::FsStorage. (We do not want to use
> the filesystem for storage, in case we need to scale ruote to another
> server). Anyway, I ran 200 requests (via thin sinatra web service) to a
> simple process definition that takes a string "Hello" and outputs via one
> participant. The FsStorage directory grew to 4.1MB. This seems like a
> lot for just inputting and outputting. This is going to be problem for
> our larger jobs that we have been writing.
I ran two tests:
---8<---
require 'rufus-json/automatic'
require 'ruote'
require 'ruote-fs'
FileUtils.rm_rf('work') rescue nil
dboard = Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
dboard.noisy = ENV['NOISY'].to_s == 'true'
dboard.register_participant 'toto' do |workitem|
print '.'
end
200.times do
wfid = dboard.launch(Ruote.define do
toto
end)
dboard.wait_for(wfid)
end
puts
puts `du -sh work/`
--->8---
Those 200 hundred terminated flows take up 12K.
Then I moved to 200 hundred un-terminated flows:
---8<---
require 'rufus-json/automatic'
require 'ruote'
require 'ruote-fs'
FileUtils.rm_rf('work') rescue nil
dboard = Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
dboard.noisy = ENV['NOISY'].to_s == 'true'
dboard.register_participant 'toto', Ruote::NullParticipant
200.times do
dboard.launch(Ruote.define do
toto
end)
end
sleep 14.0
puts
puts `du -sh work/`
--->8---
and it reached 1.6M.
> So I have 2 questions. Are the massive amount of sql requests normal, and
Yes, it's normal. Ruote needs to keep up with incoming msgs (orders) and
schedules so it polls its storage.
When activity slows down, the polling slows down a bit.
Some storage implementations (sorry, ruote-swf for example) use long polling
techniques.
If you need less polling I could adapt (or provide means to adapt the polling
frequency).
> is the amount of file system storage usual? If so, how do people work
> handle this in production environments.
Yes, it's normal. I chose to not compress at all files for the fs storage
implementation... Disk space is cheap (I guess you work with virtual
servers).
It's not too difficult to modify a storage implementation to have some degree
of compression.
Ruote keeps track of lots of redudant information (especially in
expressions), they are like many save points. It's uneconomical but easier to
implement and easier to fix.
Best regards,
--
John Mettraux - http://lambda.io/jmettraux
--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to [email protected]
to unsubscribe : send email to [email protected]
more options : http://groups.google.com/group/openwferu-users?hl=en
