Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/369#issuecomment-40034695
I tried writing a test for the Ordering parameter but am getting issues
with SparkConf not being serializable. I suspect that the surrounding context
of the Ordering parameter is causing issues. When I comment out the last two
lines that use the Ordering, the class compiles and works (though obviously
doesn't actually test anything).
```
[info] - sortByKey with explicit ordering *** FAILED *** (11 milliseconds)
[info] org.apache.spark.SparkException: Job aborted: Task not
serializable: java.io.NotSerializableException: org.apache.spark.SparkConf
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1017)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1015)
[info] at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info] at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[info] at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1015)
[info] at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:778)
[info] at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:721)
[info] at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:551)
[info] at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
[info] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
[info] at akka.actor.ActorCell.invoke(ActorCell.scala:456)
[info] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
[info] at akka.dispatch.Mailbox.run(Mailbox.scala:219)
[info] at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
[info] at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[info] at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[info] at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
```
Any ideas on how to properly call it with this extra implicit parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
