Hi,all, Thanks for your time to read this,
When I first trying to write a new Java class, and put spark in it, I always get a exception: *Exception in thread "main" org.apache.spark.SparkException: Job aborted: Task not serializable: java.io.NotSerializableException: org.dcu.test.SparkPrefix* * at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1011)* * at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1009)* * at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)* * at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)* * at org.apache.spark.scheduler.DAGScheduler.org <http://org.apache.spark.scheduler.dagscheduler.org/>$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1009)* * at org.apache.spark.scheduler.DAGScheduler.org <http://org.apache.spark.scheduler.dagscheduler.org/>$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:777)* * at org.apache.spark.scheduler.DAGScheduler.org <http://org.apache.spark.scheduler.dagscheduler.org/>$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:720)* * at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:554)* * at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$1$$anonfun$receive$1.apply(DAGScheduler.scala:201)* * at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$1$$anonfun$receive$1.apply(DAGScheduler.scala:192)* * at akka.actor.Actor$class.apply(Actor.scala:318)* * at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$1.apply(DAGScheduler.scala:173)* * at akka.actor.ActorCell.invoke(ActorCell.scala:626)* * at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)* * at akka.dispatch.Mailbox.run(Mailbox.scala:179)* * at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)* * at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)* * at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)* * at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)* * at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)* Then I found all the spark examples using main function or other static functions only, so is there any pattern in it and why?
