In the end it turns out that the issue was caused by a config settings in spark-defaults.conf. After removing this setting
spark.files.userClassPathFirst true things are back to normal. Just reporting in case f someone will have the same issue. - Gurvinder On 07/03/2014 06:49 PM, Gurvinder Singh wrote: > Just to provide more information on this issue. It seems that SPARK_HOME > environment variable is causing the issue. If I unset the variable in > spark-class script and run in the local mode my code runs fine without > the exception. But if I run with SPARK_HOME, I get the exception > mentioned below. I could run without setting SPARK_HOME but it is not > possible to run in the cluster settings, as this tells where is spark on > worker nodes. E.g. we are using Mesos as cluster manager, thus when set > master to mesos we get the exception as SPARK_HOME is not set. > > Just to mention again the pyspark works fine as well as spark-shell, > only when we are running compiled jar it seems SPARK_HOME causes some > java run time issues that we get class cast exception. > > Thanks, > Gurvinder > On 07/01/2014 09:28 AM, Gurvinder Singh wrote: >> Hi, >> >> I am having issue in running scala example code. I have tested and able >> to run successfully python example code, but when I run the scala code I >> get this error >> >> java.lang.ClassCastException: cannot assign instance of >> org.apache.spark.examples.SparkPi$$anonfun$1 to field >> org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of >> org.apache.spark.rdd.MappedRDD >> >> I have compiled spark from the github directly and running with the >> command as >> >> spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar >> --class org.apache.spark.examples.SparkPi 5 --jars >> /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar >> >> Any suggestions will be helpful. >> >> Thanks, >> Gurvinder >> >
