It might be due to the object is nested within some class which may not be
serializable.
Also you can run the appluication using this jvm parameter to see detailed
info about serialization -Dsun.io.serialization.extendedDebugInfo=true
On Fri, Oct 17, 2014 at 4:07 PM, shahab <shahab.mok...@gmail.com> wrote:
> Hi,
>
> Probably I am missing very simple principle , but something is wrong with
> my filter,
> i get "org.apache.spark.SparkException: Task not serializable" expetion.
>
> here is my filter function:
> object OBJ {
> def f1(): Boolean = {
> var i = 1;
> for (j<-1 to 10) i = i +1;
> true;
> }
> }
>
> rdd.filter(row => OBJ.f1())
>
>
> And when I run, I get the following exception:
>
> org.apache.spark.SparkException: Task not serializable
> at
> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
> at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
> at org.apache.spark.rdd.RDD.filter(RDD.scala:282)
> .......
> Caused by: java.io.NotSerializableException: org.apache.spark.SparkConf
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> ...........
>
>
>
> best,
> /Shahab
>
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chan...@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
