Re: issue with running example code
In the end it turns out that the issue was caused by a config settings in spark-defaults.conf. After removing this setting spark.files.userClassPathFirst true things are back to normal. Just reporting in case f someone will have the same issue. - Gurvinder On 07/03/2014 06:49 PM, Gurvinder Singh wrote: Just to provide more information on this issue. It seems that SPARK_HOME environment variable is causing the issue. If I unset the variable in spark-class script and run in the local mode my code runs fine without the exception. But if I run with SPARK_HOME, I get the exception mentioned below. I could run without setting SPARK_HOME but it is not possible to run in the cluster settings, as this tells where is spark on worker nodes. E.g. we are using Mesos as cluster manager, thus when set master to mesos we get the exception as SPARK_HOME is not set. Just to mention again the pyspark works fine as well as spark-shell, only when we are running compiled jar it seems SPARK_HOME causes some java run time issues that we get class cast exception. Thanks, Gurvinder On 07/01/2014 09:28 AM, Gurvinder Singh wrote: Hi, I am having issue in running scala example code. I have tested and able to run successfully python example code, but when I run the scala code I get this error java.lang.ClassCastException: cannot assign instance of org.apache.spark.examples.SparkPi$$anonfun$1 to field org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of org.apache.spark.rdd.MappedRDD I have compiled spark from the github directly and running with the command as spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar --class org.apache.spark.examples.SparkPi 5 --jars /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar Any suggestions will be helpful. Thanks, Gurvinder
Re: issue with running example code
Just to provide more information on this issue. It seems that SPARK_HOME environment variable is causing the issue. If I unset the variable in spark-class script and run in the local mode my code runs fine without the exception. But if I run with SPARK_HOME, I get the exception mentioned below. I could run without setting SPARK_HOME but it is not possible to run in the cluster settings, as this tells where is spark on worker nodes. E.g. we are using Mesos as cluster manager, thus when set master to mesos we get the exception as SPARK_HOME is not set. Just to mention again the pyspark works fine as well as spark-shell, only when we are running compiled jar it seems SPARK_HOME causes some java run time issues that we get class cast exception. Thanks, Gurvinder On 07/01/2014 09:28 AM, Gurvinder Singh wrote: Hi, I am having issue in running scala example code. I have tested and able to run successfully python example code, but when I run the scala code I get this error java.lang.ClassCastException: cannot assign instance of org.apache.spark.examples.SparkPi$$anonfun$1 to field org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of org.apache.spark.rdd.MappedRDD I have compiled spark from the github directly and running with the command as spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar --class org.apache.spark.examples.SparkPi 5 --jars /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar Any suggestions will be helpful. Thanks, Gurvinder
issue with running example code
Hi, I am having issue in running scala example code. I have tested and able to run successfully python example code, but when I run the scala code I get this error java.lang.ClassCastException: cannot assign instance of org.apache.spark.examples.SparkPi$$anonfun$1 to field org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of org.apache.spark.rdd.MappedRDD I have compiled spark from the github directly and running with the command as spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar --class org.apache.spark.examples.SparkPi 5 --jars /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar Any suggestions will be helpful. Thanks, Gurvinder
