I know this approach, only thing is, it relies on the transformation being an RDD transfomration as well and so could be applied via foreachRDD and using the rdd context to avoid a stale context after recovery/resume. My question is how to void stale context in a DStream-only transformation such as updateStateByKey / mapWithState ?
On Tue, Feb 7, 2017 at 9:19 PM Shixiong(Ryan) Zhu <shixi...@databricks.com> wrote: > It's documented here: > http://spark.apache.org/docs/latest/streaming-programming-guide.html#accumulators-broadcast-variables-and-checkpoints > > On Tue, Feb 7, 2017 at 8:12 AM, Amit Sela <amitsel...@gmail.com> wrote: > > Hi all, > > I was wondering if anyone ever used a broadcast variable within > an updateStateByKey op. ? Using it is straight-forward but I was wondering > how it'll work after resuming from checkpoint (using the rdd.context() > trick is not possible here) ? > > Thanks, > Amit > > >
