Hi I have been using Spark for quite some time using either scala or python. I wanted to give a try to groovy through scripts for small tests.
Unfortunately I get the following exception (using that simple script <https://gist.github.com/galleon/d6540327c418aa8a479f>) Is there anything I am not doing correctly here. Thanks tog Groovy4Spark $ groovy GroovySparkWordcount.groovy class org.apache.spark.api.java.JavaRDD true true Caught: org.apache.spark.SparkException: Task not serializable org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:315) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:305) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:132) at org.apache.spark.SparkContext.clean(SparkContext.scala:1893) at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:311) at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:310) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:286) at org.apache.spark.rdd.RDD.filter(RDD.scala:310) at org.apache.spark.api.java.JavaRDD.filter(JavaRDD.scala:78) at org.apache.spark.api.java.JavaRDD$filter$0.call(Unknown Source) at GroovySparkWordcount.run(GroovySparkWordcount.groovy:27) Caused by: java.io.NotSerializableException: GroovySparkWordcount Serialization stack: - object not serializable (class: GroovySparkWordcount, value: GroovySparkWordcount@7eee6c13) - field (class: GroovySparkWordcount$1, name: this$0, type: class GroovySparkWordcount) - object (class GroovySparkWordcount$1, GroovySparkWordcount$1@15c16f19) - field (class: org.apache.spark.api.java.JavaRDD$$anonfun$filter$1, name: f$1, type: interface org.apache.spark.api.java.function.Function) - object (class org.apache.spark.api.java.JavaRDD$$anonfun$filter$1, <function1>) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:81) at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:312) ... 12 more -- PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
