You could do something like this.
def rddTrasnformationUsingBroadcast(rdd: RDD[...]): RDD[...] = {
val broadcastToUse = getBroadcast() // get the reference to a
broadcast variable, new or existing.
rdd.map { ...... } // use broadcast variable
}
dstream.transform(rddTrasnformationUsingBroadcast)
The function `rddTrasnformationUsingBroadcast` will be called at every
interval, which will call `getBroadcast` every time. That's an arbitrary
function returning a broadcast variable, so you can update what broadcast
variable to use whenever you want.
TD
On Wed, Feb 18, 2015 at 6:11 AM, aanilpala <aanilp...@gmail.com> wrote:
> I am implementing a stream learner for text classification. There are some
> single-valued parameters in my implementation that needs to be updated as
> new stream items arrive. For example, I want to change learning rate as the
> new predictions are made. However, I doubt that there is a way to broadcast
> variables after the initial broadcast. So what happens if I need to
> broadcast a variable every time I update it. If there is a way to do it or
> a
> workaround for what I want to accomplish in Spark Streaming, I'd be happy
> to
> hear about it.
>
> Thanks in advance.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Periodic-Broadcast-in-Apache-Spark-Streaming-tp21703.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
