Having gone that path, we eventually gave up. Any high resolution image cannot be processed with ImageMagick without 2-4GB of RAM. We now run our workers on EC2, documented in http://artsy.github.io/blog/2012/01/31/beyond-heroku-satellite-delayed-job-workers-on-ec2/ .
On Thu, Jun 13, 2013 at 5:20 AM, Daniel Farina <[email protected]> wrote: > On Thu, Jun 13, 2013 at 2:03 AM, Josal <[email protected]> wrote: > > What I plan is to stop the process and flag it as "pending" in my > > backoffice, making the process manually later in my machine as an > exception. > > > > I've used 2X dynos, but all the memory available is used and R14 errors > also > > are thrown. Imagemagick works this way, with much memory. I've limited > > delegating to disk cache instead of memory cache > > (http://www.imagemagick.org/script/command-line-options.php#limit, > suggested > > in some places, example here: > > http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=23090), > but > > no changes. > > Tricky. Nominally this is what Linux cgroups was invented to help > with, but I think the amount of power delegated to non-root users (and > thus, on Heroku) is minimal. > > Here are two options that come to mind: > > 1) Somehow delegate these possibly expensive processes to their own 2X > dyno, taking the computation out-of-band. e.g., a queuing strategy, > multi-app delegation (one app posting to another), or the Heroku API > (be careful with that latter one, as it's basically automated "spend > money", and one can hit api limits, also, the control plane has worse > availability than standing-processes...but it has upsides, like burst > capacity and 0-cost when there's nothing to do) > > 2) Use /proc/[self|pid]/statm or /proc/[self|pid]/status (which has > similar information, but is more human-targeted) to inspect the RSS > size of a process over time to take control over over-bloated > processes. A process could watch itself grow in this way, which would > be useful if it can do it semi-frequently enough to write a "too big, > can't do" record and give up. > > If one uses /proc/pid/statm, be aware one will have to multiply by the > system page size, which has been 4096 bytes for years but *could* > change any time, so consider 'getconf PAGESIZE' to pick up the > multiplier. > > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "Heroku Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- dB. | Moscow - Geneva - Seattle - New York code.dblock.org - @dblockdotorg <http://twitter.com/#!/dblockdotorg> - artsy.net - github/dblock <https://github.com/dblock> -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
