Oops - I guess that's in 0.8.1 (unreleased) and didn't make it in 0.8.0.
On Mon, Nov 25, 2013 at 11:08 AM, Matei Zaharia <[email protected]> wrote: > The fix for large results is in 0.8.1 actually, not 0.8.0, so it’s not out > yet. > > Matei > > On Nov 25, 2013, at 10:27 AM, Andrew Ash <[email protected]> wrote: > > Thanks for the quick response Patrick! > > The downsides of always allocating an overly large buffer make sense. I'll > keep that in mind as I tune that setting for my workload. > > Also I observed the error this past weekend on 0.8.0, though I don't > remember if it was during fetching results specifically or some other stage. > I'll try to get you a copy of that stacktrace so we have something tangible > to discuss. > > Andrew > > > On Mon, Nov 25, 2013 at 10:10 AM, Patrick Wendell <[email protected]> > wrote: >> >> Good question, I think inside of akka they will allocate a buffer of >> this size for every message. So if you set it super high you'll waste >> some memory temporarily allocating these buffers. >> >> The main issue with this IIRC was for fetching results, which we fixed >> in 0.8.0 to use a different communication library. >> >> - Patrick >> >> On Mon, Nov 25, 2013 at 9:29 AM, Andrew Ash <[email protected]> wrote: >> > There have been a number of threads on this list about needing to set >> > spark.akka.frameSize to something higher than the default. The issue >> > seems >> > to come up most when one key in a groupByKey has particularly large >> > amounts >> > of data. >> > >> > What is the downside to setting this configuration parameter to the >> > maximum >> > value by default? >> > >> > Andrew > > >
