[
https://issues.apache.org/jira/browse/SPARK-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177766#comment-14177766
]
Sean Owen commented on SPARK-4018:
----------------------------------
Hm, I mean I suppose it goes without saying that the Java unit tests show that
this does in general work, and I am successfully using JavaRDD all day myself
with map and reduce.
I think the anonymous function here is the context cleaner function perhaps,
but I think that's a red herring.
I noticed I don't see a org.apache.spark.SparkContext$$anonfun$26 in the byte
code when built from master and this reminds me of a potential explanation. Are
you building against a different version of Spark than you're running?
anonymous function 26 could be something totally different at runtime if so.
That would explain why you aren't seeing any problem at compile time. I imagine
it's something like this and not a problem with Spark per se.
> RDD.reduce failing with java.lang.ClassCastException:
> org.apache.spark.SparkContext$$anonfun$26 cannot be cast to scala.Function2
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-4018
> URL: https://issues.apache.org/jira/browse/SPARK-4018
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.1
> Reporter: Haithem Turki
> Priority: Critical
>
> Hey all,
> A simple reduce operation against Spark 1.1.1 is giving me following
> exception:
> {code}
> 14/10/20 16:27:22 ERROR executor.Executor: Exception in task 9.7 in stage 0.0
> (TID 1001)
> java.lang.ClassCastException: org.apache.spark.SparkContext$$anonfun$26
> cannot be cast to scala.Function2
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:54)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> My code is a relatively simple map-reduce:
> {code}
> Map<String, Foo> aggregateTracker = rdd.map(new MapFunction(list))
> .reduce(new ReduceFunction());
> {code}
> Where:
> - MapFunction is of type Function<Record, Map<String, Object>
> - ReduceFunction is of type Function2<Map<String, Foo>, Map<String, Foo>,
> Map<String, Foo>>
> - list is just a list of Foo2
> Both Foo1 and Foo2 are serializable
> I've tried this with both the Java and Scala API, lines for each are:
> org.apache.spark.api.java.JavaRDD.reduce(JavaRDD.scala:32)
> org.apache.spark.rdd.RDD.reduce(RDD.scala:861)
> The thing being flagged is always: org.apache.spark.SparkContext$$anonfun$26
> (the number doesn't change).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
