On Wed, Jul 25, 2012 at 05:30:11AM -0700, Chad wrote: > > If manage to get this working, we owe you a dinner (and more) the next time > you're in Tokyo.
Oh, you guys are in Tokyo? Where? I'm in Hiroshima. > So here's the story with Sequel. We are using 1.9.2-p290 and Sequel 3.37.0 > and mysql2 0.3.11. > > Here's a sample of the SQL traces, if we add a logger to a Sequel > connection: > > https://gist.github.com/3175854 > > The trace is on a new and clean database, with no data in it, and simply > instantiating the Route::Engine like so: > > db = Sequel.connect(db_settings) > db.loggers << OurLogging.logger > > RuoteKit.engine = Ruote::Engine.new( > Ruote::Worker.new( > Ruote::Sequel::Storage.new(db))) OK, I'll look at it in the coming days. Seems like there are too many SELECT * FROM `documents` WHERE ((`typ` = 'configurations') AND (`ide` = 'engine')) ORDER BY `rev` DESC LIMIT 1 If you have a stack trace for one of the timeouts you mentioned, that'd be much appreciated. > Even though we started no process definitions, Ruote's Engine immediately > hits the DB. But more importantly, it never stops. The log just grows and > grows and grows. Yes, as said, it polls for msgs and schedules, for those two types of data it considers the storage as a queue. Maybe your logger could ignore selects for 'typ' "configurations", "msgs" and "schedules". That'd dramatically reduce the log size. > Regarding the FsStorage, your megabyte numbers are dramatically smaller > than ours. We are basically doing nothing as well, but I'll perform your > tests.. I grant that disk-space is cheap, but there's not enough > disk-space in the world available if we keep Ruote running for a year, > without deleting some of the stuff Ruote saves on the file system. If we > have to use FsStorage, we would need to run a cron job to clean things > occassionally. What files would we need to purge? The structure of the > FsStorage is rather daunting. Well, terminated flows weigh 0K... It seems that your test involves flows that are alive and flows that have errors. As said before, I tend to save lots of redundant info, not to lose anything. The storage implementations are also quite naive (not many shortcuts for fear of backfires). > Also, when inspecting either the file system or the database, it appears > that Ruote serializes any data - in a JSON object - any data passes through > via input, or during the processes themselves. It looks like it's not a > good idea to be passing around large files inside of workitems, since they > would either be persisted in the DB or FileSystem indefinitely. Is that a > correct conclusion? Ah yes, you have to avoid passing large stuff inside of workitems. Nowadays, it's much more efficient to pass pointers to stuff (URIs for example) than stuff itself. > So lots of questions, but I'm interested to hear about the Sequel storage > issues. OK, please don't forget to give me more details about the timeouts. > Thanks in advance for helping us. We planned on deploying on Friday, but > this is looking like a deal breaker. :( :-( -- John Mettraux - http://lambda.io/jmettraux -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
