Probably a good idea to try from scratch with the MongoDB adapter. In my tests our MongoDB storage was almost exactly 2x slower than Redis, with loads of varying sizes, but there isn't a good reason for this to be the case that I can think of, so we are doing something wrong I think. (Ruote doesn't call "get_many" for expressions during the normal course of executing a flow does it? )
I spent a few more hours reading the code for the expressions this weekend, and I finally think I have a complete grasp of how everything fits together at a high level. I believe there is a way to dramatically improve the performance and scalability characteristics of Ruote without squeezing more performance out of the storage or tinkering with when an expression chooses to persist itself just by changing the way Ruote processes expressions a little bit. I'm going to play with this in my fork and if it bears fruit, I'll ask you to take a peek and see what you think. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of John Mettraux Sent: Sunday, November 20, 2011 4:23 PM To: [email protected] Subject: Re: [ruote:3307] Re: Change storage implementations in production and other questions :) On Fri, Nov 18, 2011 at 04:56:39PM -0800, Nathan wrote: > > Thank you for the detailed response. I think I'm being too aggressive > with performance goals, so I'm going to deploy things as I have them > now and see how it runs in production. Our load tests were simulating > substantially more concurrent users than we really have, so I think > we'll be ok for some time (crossing my fingers anyway). > > I do agree that the design of ruote is heavily focused on reliability > and consistency over speed, which is definitely what you want in the > kind of system ruote was designed for. While in the future it would be > nice if ruote could be optionally tuned for speed at the cost of > consistency in the face of any kind of failure specifically for real > time user facing types of systems where, and if that day ever comes I > have some thoughts, but I think we can get by just fine with a couple > more worker processes and the tweaks I've already made to the mongo > storage. Hello Nathan, at one hand of the speed spectrum there is the in-memory storage. It complicates the architecture, but one could imagine having a dedicated in-memory backed engine for transient processes and a more persistent one for more important processes. Ruote-redis should perform well (especially the latest version). I'm currently working on a [proprietary] storage that only saves the various expressions when all the work for them is done. It's nice, but it has its own issues (when is work "done", and potential loops that make it never reach "done") The dumb "save and be safe" model followed by ruote has its advantages and shouldn't feel too weak when there are humans in the loop. > (...) > > As for the mongo storage, I would love to go through the code with > you. I'm sure it could be made much better. I'm not sure how we would > do it, but I'm very open to it any time you like. You don't mind if I give a try at it from scratch ? (So I can learn MongoDB as I go). > Thanks again for your help. I may have said this before, but Ruote is > the single best supported open source project I've used. I don't know > how you find the time, but your responsiveness, detail and patience > are really a very strong feature of Ruote. I simply try to respond as quickly as possible and to have a ruote lean enough to immediately come up with test cases to validate feedback or continue the discussion. Thanks for helping other users as well like you did, much appreciated ! Cheers, -- John Mettraux - http://lambda.io/processi -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
