On Thu, Apr 10, 2014 at 12:24 PM, Andrew Ash <and...@andrewash.com> wrote: > The biggest issue I've come across is that the cluster is somewhat unstable > when under memory pressure. Meaning that if you attempt to persist an RDD > that's too big for memory, even with MEMORY_AND_DISK, you'll often still get > OOMs. I had to carefully modify some of the space tuning parameters and GC > settings to get some jobs to even finish.
Would you mind sharing some of these settings? Even just a GitHub gist would be helpful. These are the main issues I've run into as well, and memory pressure also seems to be correlated with akka timeouts, possibly because of GC pauses.
