In general threads consume more CPU while processes consume more RAM. Reducing the number of threads may not have any considerable impact on your overall memory usage. You could try a memory profiling tool to identify memory leaks in your application or maybe there is a way to restart your sidekiq master process when it begins to consume too much memory.
— Sent from Mailbox for iPhone On Mon, Nov 11, 2013 at 11:10 AM, Neil Middleton <[email protected]> wrote: > Yeah, so R14 is more of a warning than an error - your app is still running, > but much slower than normal. > I would suggest looking at what you’re loading into memory during a worker > run - I would guess that you’re loading your *entire* app and all it’s > dependencies in order to do something that I would guess is pretty simple. > - N > On 11 November 2013 at 16:06:32, Jack Royal-Gordon ([email protected]) wrote: > I have background processes running Sidekiq that run for several hours each > night. They generate a large number of R14 errors (1600 for 2 processes each > running about 6-9 hours). But it seems like the excessive memory is slight > (not more than 520MB vs the 512MB allocated), so it would not be justified to > use jumbo (1024MB) processes. > Threads require pre-loading of the runtime environment, which includes the > Rails library. If there are parts of the library that I'm not using (and > there are), I ought to be able to fine-tune the pre-load process to exclude > them, eventually achieving a point where memory usage comes within 512MB. > Determining which parts are used and which aren't can be a tedious > experience. Can I not just rely on the paging to eventually page out the > parts I'm not using, at which point no more paging will take place because > the out-of-memory pages are never requested? > I tried lowering the number of concurrent jobs per process (i.e. threads) > from 10 to 8, but it seemed like it only made up 4-5MB in excess memory and > did not affect the number of R14 errors. In order to make up for the lost > concurrency, I added another process. The overall effect was to slow > everything down (an additional 2-1/2 hours of processing to get through the > workload), so I concluded that the extra concurrency (3x8=24 vs 2x10 = 20) > put extra load on the Postgre database which became a processing bottleneck. > So my gut feeling is that changing the jobs-per-process count is not the best > way to lower memory usage. > Any other thoughts on how to solve this (or whether it really needs to be > solved) would be appreciated. > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Heroku Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Heroku Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
