On Mon, Jul 12, 2010 at 06:39:02PM -0400, Rich Meyers wrote: > > I attempted to run some workflows I built in a cloud, where disk is actually > a network filesystem. Using the filesystem storage backend I experienced > abysmal performance. I think it stems from the sheer quantity of i/o > operations that the filesystem storage backend does - I counted over 4,000 > File.open calls on a workflow that ought to have had fewer than 50 steps.
Hello Rich, So the disk is not adjacent to the ruote worker ? > Are there other storage backends that perform fewer i/o operations? The Ruote::HashStorage does no i/o operation at all, it's all in memory / transient. Don't take it as a joke, it could be a base for "persist less" storage, something with a "now is a good time to persist" tactic. If you have a unique worker, it could be interesting to disable file locking, it could half the i/o operation count. > Doing equal number of database queries should take less time but the database > is still on network filesystem so really I'd like to have much fewer i/o > operations. Each change of state of expressions (nodes in a process instance) is saved so that workers can crash and the process instances are not gone. Granted, it's really demanding on the storages. I'd suggest using a worker adjacent to its disk (file system storage). I've been working on multiple storages for ruote 2.1.x : http://ruote.rubyforge.org/configuration.html#storage You're free to pick one or to take inspiration from one to build yours (here to help). Now, if I place this in relation with your previous email (ruote as a black box and no reliable wait). Do you always need persistence ? If you can afford some of your workflows to fail, why not having a dedicated 'transient' engine (Ruote::HashStorage based) for those scripts ? Anyway, in the end, even if you don't use ruote, I'd like to say thanks for the help and the feedback so far, sorry for the time lost, -- John Mettraux - http://jmettraux.wordpress.com -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
