No, creating DF using createDataFrame won’t work:
val peopleDF = sqlContext.createDataFrame(people)
the code can be compiled but raised the same error as toDF at the line
above.
On Wed, May 13, 2015 at 6:22 PM Sebastian Alfers
toDF is not a member of RDD object
To: SLiZn Liu sliznmail...@gmail.com
Are you sure that you are submitting it correctly? Can you post the entire
command you are using to run the .jar file via spark-submit?
On Wed, May 13, 2015 at 4:07 PM, SLiZn Liu sliznmail...@gmail.com wrote:
No, creating
Additionally, after I successfully packaged the code, and submitted
via spark-submit
webcat_2.11-1.0.jar, the following error was thrown at the line where toDF()
been called:
Exception in thread main java.lang.NoSuchMethodError:
I believe what Dean Wampler was suggesting is to use the sqlContext not the
sparkContext (sc), which is where the createDataFrame function resides:
https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.sql.SQLContext
HTH.
-Todd
On Wed, May 13, 2015 at 6:00 AM, SLiZn Liu
It's the import statement Olivier showed that makes the method available.
Note that you can also use `sc.createDataFrame(myRDD)`, without the need
for the import statement. I personally prefer this approach.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
Hi User Group,
I’m trying to reproduce the example on Spark SQL Programming Guide
https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection,
and got a compile error when packaging with sbt:
[error] myfile.scala:30: value toDF is not a member of
you need to instantiate a SQLContext :
val sc : SparkContext = ...
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
Le mar. 12 mai 2015 à 12:29, SLiZn Liu sliznmail...@gmail.com a écrit :
I added `libraryDependencies += org.apache.spark % spark-sql_2.11 %
1.3.1` to `build.sbt`
Thanks folks, really appreciate all your replies! I tried each of your
suggestions and in particular, *Animesh*‘s second suggestion of *making
case class definition global* helped me getting off the trap.
Plus, I should have paste my entire code with this mail to help the
diagnose.
REGARDS,
Todd
