Hi John (again!).
So I've explored the FsStorage. I ran the same scripts that you sent, and
I get dramatically different results. While your'e script produces 12k on
finish, mine ends at 692k.
I would love to hear how I could add long-polling, that you suggested.
On Wednesday, July 25, 2012 8:30:48 PM UTC+9, John Mettraux wrote:
>
>
> On Wed, Jul 25, 2012 at 03:21:25AM -0700, Chad wrote:
> >
> > We are building a service on ruote, and have encountered some
> performance
> > problems. We originally used Route::Sequel::Storage with the mysql2
> > adapter. On deployment we noticed that ruote made massive number of sql
> > requests, producing numerous time outs.
>
> Hello Chad,
>
> do you have details? I'd like to make ruote-sequel better.
>
>
> > For comparison, we switched to Route::FsStorage. (We do not want to use
> > the filesystem for storage, in case we need to scale ruote to another
> > server). Anyway, I ran 200 requests (via thin sinatra web service) to a
> > simple process definition that takes a string "Hello" and outputs via
> one
> > participant. The FsStorage directory grew to 4.1MB. This seems like a
> > lot for just inputting and outputting. This is going to be problem for
> > our larger jobs that we have been writing.
>
> I ran two tests:
>
> ---8<---
> require 'rufus-json/automatic'
> require 'ruote'
> require 'ruote-fs'
>
> FileUtils.rm_rf('work') rescue nil
>
> dboard =
> Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
> dboard.noisy = ENV['NOISY'].to_s == 'true'
>
> dboard.register_participant 'toto' do |workitem|
> print '.'
> end
>
> 200.times do
> wfid = dboard.launch(Ruote.define do
> toto
> end)
> dboard.wait_for(wfid)
> end
>
> puts
> puts `du -sh work/`
> --->8---
>
> Those 200 hundred terminated flows take up 12K.
>
> Then I moved to 200 hundred un-terminated flows:
>
> ---8<---
> require 'rufus-json/automatic'
> require 'ruote'
> require 'ruote-fs'
>
> FileUtils.rm_rf('work') rescue nil
>
> dboard =
> Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work')))
> dboard.noisy = ENV['NOISY'].to_s == 'true'
>
> dboard.register_participant 'toto', Ruote::NullParticipant
>
> 200.times do
> dboard.launch(Ruote.define do
> toto
> end)
> end
>
> sleep 14.0
>
> puts
> puts `du -sh work/`
> --->8---
>
> and it reached 1.6M.
>
>
> > So I have 2 questions. Are the massive amount of sql requests normal,
> and
>
> Yes, it's normal. Ruote needs to keep up with incoming msgs (orders) and
> schedules so it polls its storage.
>
> When activity slows down, the polling slows down a bit.
>
> Some storage implementations (sorry, ruote-swf for example) use long
> polling
> techniques.
>
> If you need less polling I could adapt (or provide means to adapt the
> polling
> frequency).
>
>
> > is the amount of file system storage usual? If so, how do people work
> > handle this in production environments.
>
> Yes, it's normal. I chose to not compress at all files for the fs storage
> implementation... Disk space is cheap (I guess you work with virtual
> servers).
>
> It's not too difficult to modify a storage implementation to have some
> degree
> of compression.
>
> Ruote keeps track of lots of redudant information (especially in
> expressions), they are like many save points. It's uneconomical but easier
> to
> implement and easier to fix.
>
>
> Best regards,
>
> --
> John Mettraux - http://lambda.io/jmettraux
>
>
--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to [email protected]
to unsubscribe : send email to [email protected]
more options : http://groups.google.com/group/openwferu-users?hl=en
