It gets serialized once per physical container, Instead of being serialized once per task of every stage that uses it. On Sat, Feb 20, 2016 at 8:15 AM jeff saremi <jeffsar...@hotmail.com> wrote:
> Is the broadcasted variable distributed to every executor or every worker? > Now i'm more confused > I thought it was supposed to save memory by distributing it to every > worker and the executors would share that copy > > > ------------------------------ > Date: Fri, 19 Feb 2016 16:48:59 -0800 > Subject: Re: Access to broadcasted variable > From: shixi...@databricks.com > To: jeffsar...@hotmail.com > CC: user@spark.apache.org > > > The broadcasted object is serialized in driver and sent to the executors. > And in the executor, it will deserialize the bytes to get the broadcasted > object. > > On Fri, Feb 19, 2016 at 5:54 AM, jeff saremi <jeffsar...@hotmail.com> > wrote: > > could someone please comment on this? thanks > > ------------------------------ > From: jeffsar...@hotmail.com > To: user@spark.apache.org > Subject: Access to broadcasted variable > Date: Thu, 18 Feb 2016 14:44:07 -0500 > > > > I'd like to know if the broadcasted object gets serialized when accessed > by the executor during the execution of a task? > I know that it gets serialized from the driver to the worker. This > question is inside worker when executor JVM's are accessing it > > thanks > Jeff > > >
