Re: Spark 2.0 Structured Streaming: sc.parallelize in foreach sink cause Task not serializable error

The code in ForeachWriter runs on the executors, which means that you are not allowed to use the SparkContext. This is probably why you are seeing that exception. On Sun, Sep 25, 2016 at 3:20 PM, Jianshi wrote: > Dear all: > > I am trying out the new released feature of

Spark 2.0 Structured Streaming: sc.parallelize in foreach sink cause Task not serializable error

Dear all: I am trying out the new released feature of structured streaming in Spark 2.0. I use the Structured Streaming to perform windowing by event time. I can print out the result in the console. I would like to write the result to Cassandra database through the foreach sink option. I am