Are you sure that you are submitting it correctly? Can you post the entire command you are using to run the .jar file via spark-submit?
Ok, here it is: /opt/spark-1.3.1-bin-hadoop2.6/bin/spark-submit target/scala-2.11/webcat_2.11-1.0.jar However, on the server somehow I have to specify main class using spark-submit --class WebcatApp --verbose webcat_2.11-1.0.jar to make spark recognize the main class. The error and stack trace remains the same. ---------- Forwarded message --------- From: Animesh Baranawal <animeshbarana...@gmail.com> Date: Wed, May 13, 2015 at 6:49 PM Subject: Re: value toDF is not a member of RDD object To: SLiZn Liu <sliznmail...@gmail.com> Are you sure that you are submitting it correctly? Can you post the entire command you are using to run the .jar file via spark-submit? On Wed, May 13, 2015 at 4:07 PM, SLiZn Liu <sliznmail...@gmail.com> wrote: > No, creating DF using createDataFrame won’t work: > > val peopleDF = sqlContext.createDataFrame(people) > > the code can be compiled but raised the same error as toDF at the line > above. > > On Wed, May 13, 2015 at 6:22 PM Sebastian Alfers > [sebastian.alf...@googlemail.com](mailto:sebastian.alf...@googlemail.com) > <http://mailto:%5bsebastian.alf...@googlemail.com%5D(mailto:sebastian.alf...@googlemail.com)> > wrote: > > I use: >> >> val conf = new SparkConf()... >> val sc = new SparkContext(conf) >> val sqlContext = new org.apache.spark.sql.SQLContext(sc) >> >> val rdd: RDD[...] = ... >> val schema: StructType = ... >> >> sqlContext.createDataFrame(rdd, schema) >> >> >> >> 2015-05-13 12:00 GMT+02:00 SLiZn Liu <sliznmail...@gmail.com>: >> >>> Additionally, after I successfully packaged the code, and submitted via >>> spark-submit >>> webcat_2.11-1.0.jar, the following error was thrown at the line where >>> toDF() been called: >>> >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror; >>> at WebcatApp$.main(webcat.scala:49) >>> at WebcatApp.main(webcat.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:569) >>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> Unsurprisingly, if I remove toDF, no error occurred. >>> >>> I have moved the case class definition outside of main but inside the >>> outer object scope, and removed the provided specification in build.sbt. >>> However, when I tried *Dean Wampler*‘s suggestion of using >>> sc.createDataFrame() the compiler says this function is not a member of >>> sc, and I cannot find any reference in the latest documents. What else >>> should I try? >>> >>> REGARDS, >>> Todd Leo >>> >>> >>> On Wed, May 13, 2015 at 11:27 AM SLiZn Liu <sliznmail...@gmail.com> >>> wrote: >>> >>>> Thanks folks, really appreciate all your replies! I tried each of your >>>> suggestions and in particular, *Animesh*‘s second suggestion of *making >>>> case class definition global* helped me getting off the trap. >>>> >>>> Plus, I should have paste my entire code with this mail to help the >>>> diagnose. >>>> >>>> REGARDS, >>>> Todd Leo >>>> >>>> >>>> On Wed, May 13, 2015 at 12:10 AM Dean Wampler <deanwamp...@gmail.com> >>>> wrote: >>>> >>>>> It's the import statement Olivier showed that makes the method >>>>> available. >>>>> >>>>> Note that you can also use `sc.createDataFrame(myRDD)`, without the >>>>> need for the import statement. I personally prefer this approach. >>>>> >>>>> Dean Wampler, Ph.D. >>>>> Author: Programming Scala, 2nd Edition >>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>>>> Typesafe <http://typesafe.com> >>>>> @deanwampler <http://twitter.com/deanwampler> >>>>> http://polyglotprogramming.com >>>>> >>>>> On Tue, May 12, 2015 at 9:33 AM, Olivier Girardot <ssab...@gmail.com> >>>>> wrote: >>>>> >>>>>> you need to instantiate a SQLContext : >>>>>> val sc : SparkContext = ... >>>>>> val sqlContext = new SQLContext(sc) >>>>>> import sqlContext.implicits._ >>>>>> >>>>>> Le mar. 12 mai 2015 à 12:29, SLiZn Liu <sliznmail...@gmail.com> a >>>>>> écrit : >>>>>> >>>>>>> I added `libraryDependencies += "org.apache.spark" % >>>>>>> "spark-sql_2.11" % "1.3.1"` to `build.sbt` but the error remains. Do I >>>>>>> need >>>>>>> to import modules other than `import org.apache.spark.sql.{ Row, >>>>>>> SQLContext >>>>>>> }`? >>>>>>> >>>>>>> On Tue, May 12, 2015 at 5:56 PM Olivier Girardot <ssab...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> toDF is part of spark SQL so you need Spark SQL dependency + import >>>>>>>> sqlContext.implicits._ to get the toDF method. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Olivier. >>>>>>>> >>>>>>>> Le mar. 12 mai 2015 à 11:36, SLiZn Liu <sliznmail...@gmail.com> a >>>>>>>> écrit : >>>>>>>> >>>>>>>>> Hi User Group, >>>>>>>>> >>>>>>>>> I’m trying to reproduce the example on Spark SQL Programming Guide >>>>>>>>> <https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection>, >>>>>>>>> and got a compile error when packaging with sbt: >>>>>>>>> >>>>>>>>> [error] myfile.scala:30: value toDF is not a member of >>>>>>>>> org.apache.spark.rdd.RDD[Person] >>>>>>>>> [error] val people = >>>>>>>>> sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p >>>>>>>>> => Person(p(0), p(1).trim.toInt)).toDF() >>>>>>>>> [error] >>>>>>>>> ^ >>>>>>>>> [error] one error found >>>>>>>>> [error] (compile:compileIncremental) Compilation failed >>>>>>>>> [error] Total time: 3 s, completed May 12, 2015 4:11:53 PM >>>>>>>>> >>>>>>>>> I double checked my code includes import sqlContext.implicits._ >>>>>>>>> after reading this post >>>>>>>>> <https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3c1426522113299-22083.p...@n3.nabble.com%3E> >>>>>>>>> on spark mailing list, even tried to use toDF("col1", "col2") >>>>>>>>> suggested by Xiangrui Meng in that post and got the same error. >>>>>>>>> >>>>>>>>> The Spark version is specified in build.sbt file as follows: >>>>>>>>> >>>>>>>>> scalaVersion := "2.11.6" >>>>>>>>> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % >>>>>>>>> "1.3.1" % "provided" >>>>>>>>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % >>>>>>>>> "1.3.1" >>>>>>>>> >>>>>>>>> Anyone have ideas the cause of this error? >>>>>>>>> >>>>>>>>> REGARDS, >>>>>>>>> Todd Leo >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >> >
