Hi User Group,
I’m trying to reproduce the example on Spark SQL Programming Guide
<https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection>,
and got a compile error when packaging with sbt:
[error] myfile.scala:30: value toDF is not a member of
org.apache.spark.rdd.RDD[Person]
[error] val people =
sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p
=> Person(p(0), p(1).trim.toInt)).toDF()
[error]
^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 3 s, completed May 12, 2015 4:11:53 PM
I double checked my code includes import sqlContext.implicits._ after
reading this post
<https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3c1426522113299-22083.p...@n3.nabble.com%3E>
on spark mailing list, even tried to use toDF("col1", "col2") suggested by
Xiangrui Meng in that post and got the same error.
The Spark version is specified in build.sbt file as follows:
scalaVersion := "2.11.6"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" %
"1.3.1" % "provided"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" %
"1.3.1"
Anyone have ideas the cause of this error?
REGARDS,
Todd Leo
