The fix for large results is in 0.8.1 actually, not 0.8.0, so it’s not out yet.
Matei On Nov 25, 2013, at 10:27 AM, Andrew Ash <[email protected]> wrote: > Thanks for the quick response Patrick! > > The downsides of always allocating an overly large buffer make sense. I'll > keep that in mind as I tune that setting for my workload. > > Also I observed the error this past weekend on 0.8.0, though I don't remember > if it was during fetching results specifically or some other stage. I'll try > to get you a copy of that stacktrace so we have something tangible to discuss. > > Andrew > > > On Mon, Nov 25, 2013 at 10:10 AM, Patrick Wendell <[email protected]> wrote: > Good question, I think inside of akka they will allocate a buffer of > this size for every message. So if you set it super high you'll waste > some memory temporarily allocating these buffers. > > The main issue with this IIRC was for fetching results, which we fixed > in 0.8.0 to use a different communication library. > > - Patrick > > On Mon, Nov 25, 2013 at 9:29 AM, Andrew Ash <[email protected]> wrote: > > There have been a number of threads on this list about needing to set > > spark.akka.frameSize to something higher than the default. The issue seems > > to come up most when one key in a groupByKey has particularly large amounts > > of data. > > > > What is the downside to setting this configuration parameter to the maximum > > value by default? > > > > Andrew >
