I think the behavior is by designed. Because if b is not persisted, and in each
call b.collect, broadcasted has point to a new broadcasted variable, serialized
by driver, and fetched by executors.
If you do persist, you don’t expect the RDD get changed due to new broadcasted
variable.
Thanks.
Zhan Zhang
On Aug 18, 2014, at 11:26 AM, Peng Cheng wrote:
> I'm curious to see that if you declare broadcasted wrapper as a var, and
> overwrite it in the driver program, the modification can have stable impact
> on all transformations/actions defined BEFORE the overwrite but was executed
> lazily AFTER the overwrite:
>
> val a = sc.parallelize(1 to 10)
>
>var broadcasted = sc.broadcast("broad")
>
>val b = a.map(_ + broadcasted.value)
> // b.persist()
>for (line <- b.collect()) { print(line) }
>
>println("\n===")
>broadcasted = sc.broadcast("cast")
>
>for (line <- b.collect()) { print(line) }
>
> the result is:
>
> 1broad2broad3broad4broad5broad6broad7broad8broad9broad10broad
> ===
> 1cast2cast3cast4cast5cast6cast7cast8cast9cast10cast
>
> Of course, if you persist b before overwriting it will still get the
> non-surprising result (both are 10broad... because they are persisted). This
> can be useful sometimes but may cause confusion at other times (people can
> no longer add persist at will just for backup because it may change the
> result).
>
> So far I've found no documentation supporting this feature. So can some one
> confirm that its a feature craftly designed?
>
> Yours Peng
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
