Considering the @transient annotations and the work done in the instance
initializer, not much state is really be broadcast to the executors. It
might be simpler to just create these instances on the executors, rather
than trying to broadcast them?
--
View this message in context:
http://apache
Streaming checkpoint doesn't support Accumulator or Broadcast. See
https://issues.apache.org/jira/browse/SPARK-5206
Here is a workaround:
https://issues.apache.org/jira/browse/SPARK-5206?focusedCommentId=14506806&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1450680
I prepared simple example helping in reproducing problem:
https://github.com/alberskib/spark-streaming-broadcast-issue
I think that in that way it will be easier for you to understand problem
and find solution (if any exists)
Thanks
Bartek
2015-12-16 23:34 GMT+01:00 Bartłomiej Alberski :
> Fir
First of all , thanks @tdas for looking into my problem.
Yes, I checked it seperately and it is working fine. For below piece of
code there is no single exception and values are sent correctly.
val reporter = new MyClassReporter(...)
reporter.send(...)
val out = new FileOutputStream("
Could you test serializing and deserializing the MyClassReporter class
separately?
On Mon, Dec 14, 2015 at 8:57 AM, Bartłomiej Alberski
wrote:
> Below is the full stacktrace(real names of my classes were changed) with
> short description of entries from my code:
>
> rdd.mapPartitions{ partition
Below is the full stacktrace(real names of my classes were changed) with
short description of entries from my code:
rdd.mapPartitions{ partition => //this is the line to which second
stacktrace entry is pointing
val sender = broadcastedValue.value // this is the maing place to which
first stack
Can you show the complete stack trace for the ClassCastException ?
Please see the following thread:
http://search-hadoop.com/m/q3RTtgEUHVmJA1T1
Cheers
On Mon, Dec 14, 2015 at 7:33 AM, alberskib wrote:
> Hey all,
>
> When my streaming application is restarting from failure (from checkpoint)
> I
