Wow! Thanks for the speedy and helpful responses Mario and John. I had seen the discussion about the WaitLogger and had mistakenly thought that was only related to "noisy" mode. I'll take a look and make some adjustments there.
Regarding threading, we are currently returning true from #do_not_thread in all participants until we can determine why we were receiving these errors. The specific error we received was "ThreadError: can't create Thread (11)". I'm still under the impression that this was due to the fact that we were out of memory. After we make some of these other changes we'll have to do some more experimenting with this option. The good news is Ruote caught these errors and we were able to use Dashboard#replay_at_error to successfully restart these workflows. :D Thanks for the zlib commit! We will definitely try this out. I wasn't sure if the issue was in terms of speed (encode/decode) or size/persistance. I imagine the combination yajl-ruby and your zlib update will make a dent in both! I will be working most of this week so I'll make incremental changes and report back on my progress. Thanks again to you both, happy holidays! On Sunday, December 23, 2012 8:53:49 PM UTC-5, John Mettraux wrote: > > > > On Sun, Dec 23, 2012 at 12:21 AM, John Allen > > <[email protected]<javascript:>> > wrote: > > > > > First of all we want thank you, John Mettraux, for your outstanding > work > > > on Ruote and your seemingly tireless responses to peoples questions > and > > > issues! > > Hello John, > > many thanks, much appreciated! > > > > > We've been running an implementation of Ruote in production for about > 1.5 > > > years. The relavent parts of our stack include: > > > > > > - Heroku > > > - rails 3 > > > - ruote > > > - ruote-kit (web and workers) > > > - ruote-redis > > > > > > We use route to do marketing automation (phone calls, emails etc...). > Up > > > until recently our load has been fairly light but we've been pushing > more > > > and more workflows (generally in large bursts) through our application > and > > > have started to run into some performance issues. At the moment we > have the > > > following stats: > > > > > > - ~100k expressions > > > - ~6k schedules > > > - 400 MB redis store (actually this seems to fluctuate between 300 > and > > > 500 MB) > > > > - unknown number of expressions are abandoned by external services, > > > could these cause performance issues? > > Yes, this is possible. I should investigate having a (gasp) garbage > collector for ruote, or at least something that diagnoses the set of > expressions and hightlights orphaned ones. > > > > > We've started getting memory warnings from Heroku on our workers and > they > > > will eventually stop processing messages until we restart them. Heroku > > > provides 512 MB of memory and then starts paging to disk after that. > We've > > > seen warnings with memory usage approaching 1 GB. It's unclear to us > how a > > > worker could be consuming more than 2x in memory over storage. > > > > > > While looking through the code this week we noticed the frequent, > > > perhaps inefficient (with so many), checks for schedules to process. > > > We've drastically extended the time between checks (perhaps this could > be a > > > configuration option) as we do not have any schedules that need to be > > > processed precisely on time. This change seems to have improved things > > > dramatically but over time we still seem to get memory warnings and > > > eventually stalled workers. We've experimented with the number of > worker > > > processes and have found that we get the best performance with 5-10 > process > > > running. > > OK, I will add a configuration option for this. Thanks! > > > > > For some time we had threading disabled so we tried re-enabling that > in > > > the effort to improve performance. It's tough to tell because we > haven't > > > done formal benchmarks but it appears that we did see a boost in > > > performance however we eventually started receiving errors as Ruote > tried > > > to create new threads. Our theory is this is related to the lack of > memory > > > on the virtual. > > By threading disabled, do you mean you made your participant return true > in > their #do_not_thread method? Or by having set > 'participant_threads_enabled' > => false globally? > > > > > We tend to have fairly simple workflows with a participant that > doesn't > > > reply until it gets more information and then a few more participants > that > > > reply fairly quickly in a cursor. We're not setting any variables and > tend > > > to have a few :if conditions. > > > > > > While reading another thread on performance here ( > > > > https://groups.google.com/d/topic/openwferu-users/IQloVPNuzcI/discussion) > > > I got to thinking about the encoding and decoding of data. I realize > that > > > we do not have yajl available in our bundle and that might provide > some > > > performance improvements. > > Yes, I always use yajl-ruby. It was one of the reasons I went all JSON > with > ruote 2.0. It's very good. > > > > > I've also been investigating MessagePack ( > > > http://msgpack.org/) and thinking that with support in Redis it might > > > make an even better improvement (see benchmarks here: > > > https://gist.github.com/1159138). Should we go down that road I > wonder if > > > you have any advice on converting the existing messages. Would you add > code > > > to detect the message format and slowly transition over time or take > the > > > system offline and try to re-encode all of the existing messages? > Also, > > > does adding MessagePack to rufus-json make sense (not strictly JSON > > > related) or would you recommend overriding the #encode #decode methods > in > > > ruote-redis? > > I would rather recommend overriding the encode/decode methods in > ruote-redis > (yes, I factored them out > > > https://github.com/jmettraux/ruote-redis/commit/8ff58ff5b1630f63f41f6ac7ab2ce2389e116e40 > > > ) > > But would that help: > > > https://github.com/jmettraux/ruote-redis/commit/29fd51c223d9ff82245e4352dc245942383a31b5 > > > ? > > There can be plenty of redundancy in the JSON produced by ruote. This > commit > brings in zlib compression. I've been using it in another (proprietary) > storage happily. It cuts significantly the weight of the data pushed to > Redis. > > Note: it reads compressed and uncompressed JSON, so it's able to "live > migrate" from non compressed ruote-redis Redises. > > MessagePack looks great, but I guess its output is not as small as > compressed > json. Feel free to experiment, looking forward to your results. > > > > > Overall I wanted to get some impressions and see if others have had > > > performance and/or memory issues with the kind of load we've been > putting > > > on Ruote. I've tried to include as much detail as possible. If there > are > > > specific questions I could answer please let me know. I realize I > haven't > > > given specific benchmark information so if that would help I can try > to > > > perform some benchmarking (been too busy putting out fires up until > now). > > > On Sun, Dec 23, 2012 at 07:03:23PM +0100, Mario Camou wrote: > > > > We had some serious memory problems earlier this week. I found out that > by > > default the Ruote History is saved in-memory, and the WaitLogger (which > is > > usually used only for testing) was saving the last 147 messages > processed > > by Ruote. In our case since this specific workflow handles large > workitems, > > memory use was skyrocketing. > > Hello Mario! > > Mario refers to this thread: > > > https://groups.google.com/forum/?fromgroups#!topic/openwferu-users/Y8xCc6yVwRg > > > > I did two things: > > > > * Change the History to use the same DB as the Storage. For that, add > the > > following line: > > > > RuoteKit.engine.add_service('history', 'ruote/log/storage_history', > > 'Ruote::StorageHistory') > > Excellent point. > > I'm a bit worried for John though, they are using Redis and putting all > the > history in it will quickly make it run out of memory. > > Therefore I've made sure the default history can be tuned out with > > ---8<--- > RuoteKit.engine = > Ruote::Engine.new( > Ruote::Worker.new( > Ruote::Redis::Storage.new( > 'db' => 14, 'thread_safe' => true, 'pop_count' => 28, > 'history_max_size' => 0))) # disable default history > --->8--- > > Or use a custom history service if history has to be kept around. > > Here is the commit for 'history_max_size': > > > https://github.com/jmettraux/ruote/commit/66b188ab71067f629ff6ebbc392a12236119cfb3 > > > > > * Set the WaitLogger to save zero messages. When you create the Engine, > > pass in the option: > > > > RuoteKit.engine = Ruote::Engine.new( > > Ruote::Worker.new( > > Ruote::FsStorage.new("ruote_work_#{RuoteKit.env}"))) > > Ruote::Mon::Storage.new( > > Mongo::Connection.new(mongo_host, mongo_port)[mongo_dbname], { > > 'wait_logger_max' => 0 }))) > > Do you mean > > ---8<--- > RuoteKit.engine = > Ruote::Engine.new( > Ruote::Worker.new( > Ruote::Mon::Storage.new( > Mongo::Connection.new(mongo_host, mongo_port)[mongo_dbname], > { 'wait_logger_max' => 0 }))) > --->8--- > > ? ;-) > > > > In our case we're using RuoteMon for storage, but the same principle > > applies: you pass in the Engine options when creating the Storage. > > +1 > > > Merry Christmas to you guys, > > -- > John Mettraux - http://lambda.io/jmettraux > > -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
