By the way, you're example is using Route::Dashboard.new and mine is using Route::Engine. The Sequel problems still occurs, if I switch to the dashboard. It also occurs if I don't assign the engine to RuoteKit.
Thanks again, On Wednesday, July 25, 2012 9:30:11 PM UTC+9, Chad wrote: > > Hi John, > > If manage to get this working, we owe you a dinner (and more) the next > time you're in Tokyo. > > So here's the story with Sequel. We are using 1.9.2-p290 and Sequel > 3.37.0 and mysql2 0.3.11. > > Here's a sample of the SQL traces, if we add a logger to a Sequel > connection: > > https://gist.github.com/3175854 > > The trace is on a new and clean database, with no data in it, and simply > instantiating the Route::Engine like so: > > db = Sequel.connect(db_settings) > db.loggers << OurLogging.logger > > RuoteKit.engine = Ruote::Engine.new( > Ruote::Worker.new( > Ruote::Sequel::Storage.new(db))) > > Even though we started no process definitions, Ruote's Engine immediately > hits the DB. But more importantly, it never stops. The log just grows and > grows and grows. > > Regarding the FsStorage, your megabyte numbers are dramatically smaller > than ours. We are basically doing nothing as well, but I'll perform your > tests.. I grant that disk-space is cheap, but there's not enough > disk-space in the world available if we keep Ruote running for a year, > without deleting some of the stuff Ruote saves on the file system. If we > have to use FsStorage, we would need to run a cron job to clean things > occassionally. What files would we need to purge? The structure of the > FsStorage is rather daunting. > > Also, when inspecting either the file system or the database, it appears > that Ruote serializes any data - in a JSON object - any data passes through > via input, or during the processes themselves. It looks like it's not a > good idea to be passing around large files inside of workitems, since they > would either be persisted in the DB or FileSystem indefinitely. Is that a > correct conclusion? > > So lots of questions, but I'm interested to hear about the Sequel storage > issues. > > Thanks in advance for helping us. We planned on deploying on Friday, but > this is looking like a deal breaker. :( > > Chad > > > > On Wednesday, July 25, 2012 8:30:48 PM UTC+9, John Mettraux wrote: >> >> >> On Wed, Jul 25, 2012 at 03:21:25AM -0700, Chad wrote: >> > >> > We are building a service on ruote, and have encountered some >> performance >> > problems. We originally used Route::Sequel::Storage with the mysql2 >> > adapter. On deployment we noticed that ruote made massive number of >> sql >> > requests, producing numerous time outs. >> >> Hello Chad, >> >> do you have details? I'd like to make ruote-sequel better. >> >> >> > For comparison, we switched to Route::FsStorage. (We do not want to >> use >> > the filesystem for storage, in case we need to scale ruote to another >> > server). Anyway, I ran 200 requests (via thin sinatra web service) to >> a >> > simple process definition that takes a string "Hello" and outputs via >> one >> > participant. The FsStorage directory grew to 4.1MB. This seems like >> a >> > lot for just inputting and outputting. This is going to be problem >> for >> > our larger jobs that we have been writing. >> >> I ran two tests: >> >> ---8<--- >> require 'rufus-json/automatic' >> require 'ruote' >> require 'ruote-fs' >> >> FileUtils.rm_rf('work') rescue nil >> >> dboard = >> Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work'))) >> dboard.noisy = ENV['NOISY'].to_s == 'true' >> >> dboard.register_participant 'toto' do |workitem| >> print '.' >> end >> >> 200.times do >> wfid = dboard.launch(Ruote.define do >> toto >> end) >> dboard.wait_for(wfid) >> end >> >> puts >> puts `du -sh work/` >> --->8--- >> >> Those 200 hundred terminated flows take up 12K. >> >> Then I moved to 200 hundred un-terminated flows: >> >> ---8<--- >> require 'rufus-json/automatic' >> require 'ruote' >> require 'ruote-fs' >> >> FileUtils.rm_rf('work') rescue nil >> >> dboard = >> Ruote::Dashboard.new(Ruote::Worker.new(Ruote::FsStorage.new('work'))) >> dboard.noisy = ENV['NOISY'].to_s == 'true' >> >> dboard.register_participant 'toto', Ruote::NullParticipant >> >> 200.times do >> dboard.launch(Ruote.define do >> toto >> end) >> end >> >> sleep 14.0 >> >> puts >> puts `du -sh work/` >> --->8--- >> >> and it reached 1.6M. >> >> >> > So I have 2 questions. Are the massive amount of sql requests normal, >> and >> >> Yes, it's normal. Ruote needs to keep up with incoming msgs (orders) and >> schedules so it polls its storage. >> >> When activity slows down, the polling slows down a bit. >> >> Some storage implementations (sorry, ruote-swf for example) use long >> polling >> techniques. >> >> If you need less polling I could adapt (or provide means to adapt the >> polling >> frequency). >> >> >> > is the amount of file system storage usual? If so, how do people work >> > handle this in production environments. >> >> Yes, it's normal. I chose to not compress at all files for the fs storage >> implementation... Disk space is cheap (I guess you work with virtual >> servers). >> >> It's not too difficult to modify a storage implementation to have some >> degree >> of compression. >> >> Ruote keeps track of lots of redudant information (especially in >> expressions), they are like many save points. It's uneconomical but >> easier to >> implement and easier to fix. >> >> >> Best regards, >> >> -- >> John Mettraux - http://lambda.io/jmettraux >> >> -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
