Hello,
When I run a spark job with spark-submit it fails with below exception for
code line
/*val webLogDF = webLogRec.toDF().select("ip", "date", "name"")*/
I had similar issue running from spark-shell, then realized that I needed
sqlContext.implicit._
Now my code has the following imports
/*
import org.apache.spark._
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
*/
Code works fine from spark-shell REPL. It also runs fine when run in local
mode from Eclipse. I get this
error only when I submit to cluster using spark-submit.
bin/spark-submit /local/weblog-analysis_2.11-1.0.jar --class WebLogAnalysis
--master spark://machu:7077
I'm testing with spark 1.4. My code was built using scala 2.11 and
spark+sparkSQL 1.4.0 as dependency in build.sbt
Exception in thread "main" java.lang.NoSuchMethodError:
scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
at WebLogAnalysis$.readWebLogFiles(WebLogAnalysis.scala:38)
at WebLogAnalysis$.main(WebLogAnalysis.scala:62)
at WebLogAnalysis.main(WebLogAnalysis.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I can provide more code or log if that will help. Let me know.
Srikanth
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-RDD-to-DF-fails-with-toDF-tp23499.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
