Thanks for the quick response Patrick! The downsides of always allocating an overly large buffer make sense. I'll keep that in mind as I tune that setting for my workload.
Also I observed the error this past weekend on 0.8.0, though I don't remember if it was during fetching results specifically or some other stage. I'll try to get you a copy of that stacktrace so we have something tangible to discuss. Andrew On Mon, Nov 25, 2013 at 10:10 AM, Patrick Wendell <[email protected]>wrote: > Good question, I think inside of akka they will allocate a buffer of > this size for every message. So if you set it super high you'll waste > some memory temporarily allocating these buffers. > > The main issue with this IIRC was for fetching results, which we fixed > in 0.8.0 to use a different communication library. > > - Patrick > > On Mon, Nov 25, 2013 at 9:29 AM, Andrew Ash <[email protected]> wrote: > > There have been a number of threads on this list about needing to set > > spark.akka.frameSize to something higher than the default. The issue > seems > > to come up most when one key in a groupByKey has particularly large > amounts > > of data. > > > > What is the downside to setting this configuration parameter to the > maximum > > value by default? > > > > Andrew >
