On Wed, Nov 23, 2011 at 05:10:37PM -0800, Nathan Stults wrote: > > Awesome work. I removed all my elaborate locking, which I had been > blindly trying to copy from the Redis adapter, and used your pure > optimistic concurrency approach instead for a noticeable speed boost.
Hello Nathan, glad it's useful. After this work on ruote-mon, I'm tempted to make it the official MongoDB ruote storage (by official I mean, I would package/ci/release/support it). I have to say that since I worked on it, I'm interested in learning more about MongoDB. What do you think ? > I finished my proof of concept on an optimized storage implementation > and was partially successful. My approach was to create a storage that > would fully execute an incoming message and any messages resulting from > the application of that message persisting to in memory storage in a > sort of transaction or "unit of work" and then persist to the DB at the > end of the transaction. Concurrency is handled by locking on the WFID > rather than the expression id, and a transaction is processed without > stopping until it is complete. "Complete" means that if the expression > initially processed caused any other messages to be placed on the queue, > those are all processed as well, recursively, until no more messages are > generated (i.e. no more expressions will reply). As I wrote earlier on the thread I use this approach for a proprietary storage implementation. It keeps executing the msgs. It's "complete" (I used the term "done"), when all the msgs have executed or all msgs are "dispatch" calls. There is a weak point: if you run into a loop, "complete" won't be reached. In our proprietary storage we have to peg the number of messages processes to prevent the worker from dedicating itself fully to the looping flow (thus becoming somehow a zombie worker). I wonder if your approach is really concurrent: it's always the same worker processing all the msgs until "complete". > This approach was effective, but not dramatically so. My load test > launches 10 simultaneous workflows, replies a work item, and waits for > five more items from a concurrence. I was able to improve the > performance of this environment of the Mongo storage by about 50%, and > the Redis storage by about 30%. These percentages were consistent (even > a little better) when the load was 100 simultaneous workflows instead of > ten. > > One interesting side effect of this work is that the characteristics of > Ruote's response to load are different. Originally, due to the strictly > round robin nature of putting fine grained, atomic components of a > single logical method call into a fifo queue, all work would always be > completed at roughly the same time. If the load was high, the work would > take a long time to complete, but complete pretty much all at once. With > the alternative techniques, work is completed in a first come first > serve fashion, and the workflows tended to complete one at a time > throughout the life of the test. > > I was expecting more striking results, but the result is good enough for > our operations. This technique will allow us to comfortably stay with > Mongo DB for quite some time, keeping Redis in reserve if we need > another boost. Redis processed 10 workflows using 2 workers in about 10 > seconds using my hybrid storage, and Mongo DB took an average of 14 > seconds. By contrast, without the hybrid storage Redis averaged 15 > seconds and Mongo DB 30 seconds). All mongo tests used your improved > approach to concurrency. > > The experiment wasn't effective enough that I intend to package this > stuff up or make it generally available, but if anyone else is ever in > our shoes I wouldn't mind doing so upon request. I'm really happy someone worked so deeply on ruote, it's been a while. Great work ! -- John Mettraux - http://lambda.io/processi -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
