You know that's an interesting point about the single worker; has anyone ever used Skynet? Their fortay is distributing map/reduce problems in the background, but they also mixed in a function to ActiveRecord called send_later which allowed you to send off any method of a model for later processing. I wrote my own mixin that does the same thing with Backgroundrb and a GenericWorker class (and I added send_now_or_later so I could have a fallback) because I liked keeping the functions on the model. That would lend itself to reducing the backgroundrb overhead as you'd have a single worker for any asynchronous AR processing, although it doesn't help with the cron-style jobs.
\Peter -----Original Message----- From: Adam Williams [mailto:[email protected]] Sent: Monday, August 10, 2009 6:50 PM To: Peter Gengler Cc: [email protected] Subject: Re: [Backgroundrb-devel] Question about backgroundrb and resource usage On Aug 10, 2009, at 3:44 PM, Peter Gengler wrote: > So I have a somewhat general question about backgroundrb and > resource usage. I hadn't really considered the fact that it spools > up a new rails process for each worker until I started making more > workers; now that I'm have half a dozen it seems like the current > design is really unscalable. I.e. if my rails stack takes 200MB of > memory for example, that means 10 workers use 2GB of memory? That > seems overly excessive to me, am I missing something that prevents > that resource bloat? As far as I have seen, you're not missing anything. I had the same problem and ended up having a single worker, with different methods for the work. I'm considering getting rid of backgroundrb altogether, favoring instead simply loading ActiveRecord and the model classes I need, with the hope that a cron task can fire the thing up within a few seconds. We need tasks to run every minute, so it must be fast loading. Anyway, that would allow us to unload a couple hundred meg at Engine Yard. adam _______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
