> On Sun, Dec 23, 2012 at 12:21 AM, John Allen <[email protected]> wrote: > > > First of all we want thank you, John Mettraux, for your outstanding work > > on Ruote and your seemingly tireless responses to peoples questions and > > issues!
Hello John, many thanks, much appreciated! > > We've been running an implementation of Ruote in production for about 1.5 > > years. The relavent parts of our stack include: > > > > - Heroku > > - rails 3 > > - ruote > > - ruote-kit (web and workers) > > - ruote-redis > > > > We use route to do marketing automation (phone calls, emails etc...). Up > > until recently our load has been fairly light but we've been pushing more > > and more workflows (generally in large bursts) through our application and > > have started to run into some performance issues. At the moment we have the > > following stats: > > > > - ~100k expressions > > - ~6k schedules > > - 400 MB redis store (actually this seems to fluctuate between 300 and > > 500 MB) > > - unknown number of expressions are abandoned by external services, > > could these cause performance issues? Yes, this is possible. I should investigate having a (gasp) garbage collector for ruote, or at least something that diagnoses the set of expressions and hightlights orphaned ones. > > We've started getting memory warnings from Heroku on our workers and they > > will eventually stop processing messages until we restart them. Heroku > > provides 512 MB of memory and then starts paging to disk after that. We've > > seen warnings with memory usage approaching 1 GB. It's unclear to us how a > > worker could be consuming more than 2x in memory over storage. > > > > While looking through the code this week we noticed the frequent, > > perhaps inefficient (with so many), checks for schedules to process. > > We've drastically extended the time between checks (perhaps this could be a > > configuration option) as we do not have any schedules that need to be > > processed precisely on time. This change seems to have improved things > > dramatically but over time we still seem to get memory warnings and > > eventually stalled workers. We've experimented with the number of worker > > processes and have found that we get the best performance with 5-10 process > > running. OK, I will add a configuration option for this. Thanks! > > For some time we had threading disabled so we tried re-enabling that in > > the effort to improve performance. It's tough to tell because we haven't > > done formal benchmarks but it appears that we did see a boost in > > performance however we eventually started receiving errors as Ruote tried > > to create new threads. Our theory is this is related to the lack of memory > > on the virtual. By threading disabled, do you mean you made your participant return true in their #do_not_thread method? Or by having set 'participant_threads_enabled' => false globally? > > We tend to have fairly simple workflows with a participant that doesn't > > reply until it gets more information and then a few more participants that > > reply fairly quickly in a cursor. We're not setting any variables and tend > > to have a few :if conditions. > > > > While reading another thread on performance here ( > > https://groups.google.com/d/topic/openwferu-users/IQloVPNuzcI/discussion) > > I got to thinking about the encoding and decoding of data. I realize that > > we do not have yajl available in our bundle and that might provide some > > performance improvements. Yes, I always use yajl-ruby. It was one of the reasons I went all JSON with ruote 2.0. It's very good. > > I've also been investigating MessagePack ( > > http://msgpack.org/) and thinking that with support in Redis it might > > make an even better improvement (see benchmarks here: > > https://gist.github.com/1159138). Should we go down that road I wonder if > > you have any advice on converting the existing messages. Would you add code > > to detect the message format and slowly transition over time or take the > > system offline and try to re-encode all of the existing messages? Also, > > does adding MessagePack to rufus-json make sense (not strictly JSON > > related) or would you recommend overriding the #encode #decode methods in > > ruote-redis? I would rather recommend overriding the encode/decode methods in ruote-redis (yes, I factored them out https://github.com/jmettraux/ruote-redis/commit/8ff58ff5b1630f63f41f6ac7ab2ce2389e116e40 ) But would that help: https://github.com/jmettraux/ruote-redis/commit/29fd51c223d9ff82245e4352dc245942383a31b5 ? There can be plenty of redundancy in the JSON produced by ruote. This commit brings in zlib compression. I've been using it in another (proprietary) storage happily. It cuts significantly the weight of the data pushed to Redis. Note: it reads compressed and uncompressed JSON, so it's able to "live migrate" from non compressed ruote-redis Redises. MessagePack looks great, but I guess its output is not as small as compressed json. Feel free to experiment, looking forward to your results. > > Overall I wanted to get some impressions and see if others have had > > performance and/or memory issues with the kind of load we've been putting > > on Ruote. I've tried to include as much detail as possible. If there are > > specific questions I could answer please let me know. I realize I haven't > > given specific benchmark information so if that would help I can try to > > perform some benchmarking (been too busy putting out fires up until now). On Sun, Dec 23, 2012 at 07:03:23PM +0100, Mario Camou wrote: > > We had some serious memory problems earlier this week. I found out that by > default the Ruote History is saved in-memory, and the WaitLogger (which is > usually used only for testing) was saving the last 147 messages processed > by Ruote. In our case since this specific workflow handles large workitems, > memory use was skyrocketing. Hello Mario! Mario refers to this thread: https://groups.google.com/forum/?fromgroups#!topic/openwferu-users/Y8xCc6yVwRg > I did two things: > > * Change the History to use the same DB as the Storage. For that, add the > following line: > > RuoteKit.engine.add_service('history', 'ruote/log/storage_history', > 'Ruote::StorageHistory') Excellent point. I'm a bit worried for John though, they are using Redis and putting all the history in it will quickly make it run out of memory. Therefore I've made sure the default history can be tuned out with ---8<--- RuoteKit.engine = Ruote::Engine.new( Ruote::Worker.new( Ruote::Redis::Storage.new( 'db' => 14, 'thread_safe' => true, 'pop_count' => 28, 'history_max_size' => 0))) # disable default history --->8--- Or use a custom history service if history has to be kept around. Here is the commit for 'history_max_size': https://github.com/jmettraux/ruote/commit/66b188ab71067f629ff6ebbc392a12236119cfb3 > * Set the WaitLogger to save zero messages. When you create the Engine, > pass in the option: > > RuoteKit.engine = Ruote::Engine.new( > Ruote::Worker.new( > Ruote::FsStorage.new("ruote_work_#{RuoteKit.env}"))) > Ruote::Mon::Storage.new( > Mongo::Connection.new(mongo_host, mongo_port)[mongo_dbname], { > 'wait_logger_max' => 0 }))) Do you mean ---8<--- RuoteKit.engine = Ruote::Engine.new( Ruote::Worker.new( Ruote::Mon::Storage.new( Mongo::Connection.new(mongo_host, mongo_port)[mongo_dbname], { 'wait_logger_max' => 0 }))) --->8--- ? ;-) > In our case we're using RuoteMon for storage, but the same principle > applies: you pass in the Engine options when creating the Storage. +1 Merry Christmas to you guys, -- John Mettraux - http://lambda.io/jmettraux -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
