i see the top method in RDD class , you can use this method to get top N , but
i found some error when i use this method .... it's seem as when mapPartitions
in top , the task result may be an Nil , then reduce will result an class cast
exception as blow :java.lang.ClassCastException:
scala.collection.immutable.Nil$ cannot be cast to
org.apache.spark.util.BoundedPriorityQueue
at org.apache.spark.rdd.RDD$$anonfun$top$2.apply(RDD.scala:873)
at org.apache.spark.rdd.RDD$$anonfun$6.apply(RDD.scala:671)
at org.apache.spark.rdd.RDD$$anonfun$6.apply(RDD.scala:668)
at org.apache.spark.scheduler.JobWaiter.taskSucceeded(JobWaiter.scala:56)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:859)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:616)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
my spark version is : 0.9.0 , i don't konw is this a bug of spark ..
there are another method to get top N , you can use sortByKey method and then
use take method to get the top N value of your RDD
mailforledkk
From: Jeyaraj, Arockia R (Arockia)Date: 2014-04-09 21:55To:
user@spark.apache.orgSubject: To Ten RDDHi , Can you any one tell how to get
Top ten RDD by value? ThanksArockia Raja
