Please do use your new Mongo storage as the official one. Ours is very much oriented around our specific needs.
> There is a weak point: if you run into a loop, "complete" won't be reached. > In our proprietary storage we have to peg the number of messages processes to prevent the worker from dedicating itself fully to the looping flow (thus becoming somehow a zombie worker). Do you mean you have workflows with an infinite loop and no dispatches in the loop? Or do you just have huge iterators iterating very large lists? Could you show an example? Our human-centric workflows don't tend to have infinite loops without some kind of dispatch or timeout to release the worker, but I might not be understanding what you are saying. > I wonder if your approach is really concurrent: it's always the same worker processing all the msgs until "complete". Well, true, the worker itself isn't concurrent, concurrency is achieved through multiple workers, but this is more or less the way it was before except a little more efficient for our use cases. However, this assumes our style of workflows which are always "human provides input; ruote calculates the next set of work and dispatches work items to AMQP; rinse; repeat". Because the work ruote has to do in a single transaction is always finite and usually pretty small we didn't really lose any practical concurrency, even with a single worker. It sounds like your use cases involve much longer running, fully automated workflows, which wouldn't work nearly as well using the "unit of work" style for the reasons you already mentioned. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of John Mettraux Sent: Wednesday, November 23, 2011 5:50 PM To: [email protected] Subject: Re: [ruote:3313] Re: Change storage implementations in production and other questions :) On Wed, Nov 23, 2011 at 05:10:37PM -0800, Nathan Stults wrote: > > Awesome work. I removed all my elaborate locking, which I had been > blindly trying to copy from the Redis adapter, and used your pure > optimistic concurrency approach instead for a noticeable speed boost. Hello Nathan, glad it's useful. After this work on ruote-mon, I'm tempted to make it the official MongoDB ruote storage (by official I mean, I would package/ci/release/support it). I have to say that since I worked on it, I'm interested in learning more about MongoDB. What do you think ? > I finished my proof of concept on an optimized storage implementation > and was partially successful. My approach was to create a storage that > would fully execute an incoming message and any messages resulting > from the application of that message persisting to in memory storage > in a sort of transaction or "unit of work" and then persist to the DB > at the end of the transaction. Concurrency is handled by locking on > the WFID rather than the expression id, and a transaction is processed > without stopping until it is complete. "Complete" means that if the > expression initially processed caused any other messages to be placed > on the queue, those are all processed as well, recursively, until no > more messages are generated (i.e. no more expressions will reply). As I wrote earlier on the thread I use this approach for a proprietary storage implementation. It keeps executing the msgs. It's "complete" (I used the term "done"), when all the msgs have executed or all msgs are "dispatch" calls. There is a weak point: if you run into a loop, "complete" won't be reached. In our proprietary storage we have to peg the number of messages processes to prevent the worker from dedicating itself fully to the looping flow (thus becoming somehow a zombie worker). I wonder if your approach is really concurrent: it's always the same worker processing all the msgs until "complete". > This approach was effective, but not dramatically so. My load test > launches 10 simultaneous workflows, replies a work item, and waits for > five more items from a concurrence. I was able to improve the > performance of this environment of the Mongo storage by about 50%, and > the Redis storage by about 30%. These percentages were consistent > (even a little better) when the load was 100 simultaneous workflows > instead of ten. > > One interesting side effect of this work is that the characteristics > of Ruote's response to load are different. Originally, due to the > strictly round robin nature of putting fine grained, atomic components > of a single logical method call into a fifo queue, all work would > always be completed at roughly the same time. If the load was high, > the work would take a long time to complete, but complete pretty much > all at once. With the alternative techniques, work is completed in a > first come first serve fashion, and the workflows tended to complete > one at a time throughout the life of the test. > > I was expecting more striking results, but the result is good enough > for our operations. This technique will allow us to comfortably stay > with Mongo DB for quite some time, keeping Redis in reserve if we > need another boost. Redis processed 10 workflows using 2 workers in > about 10 seconds using my hybrid storage, and Mongo DB took an average > of 14 seconds. By contrast, without the hybrid storage Redis averaged > 15 seconds and Mongo DB 30 seconds). All mongo tests used your > improved approach to concurrency. > > The experiment wasn't effective enough that I intend to package this > stuff up or make it generally available, but if anyone else is ever in > our shoes I wouldn't mind doing so upon request. I'm really happy someone worked so deeply on ruote, it's been a while. Great work ! -- John Mettraux - http://lambda.io/processi -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
