Here is the specific code
val sc = new SparkContext(s"local[$NWorkers]", "HBaseTestsSparkContext")
val ctx = new SQLContext(sc)
import ctx._
case class MyTable(col1: String, col2: Byte)
val myRows = ctx.sparkContext.parallelize((Range(1,21).map{ix =>
MyTable(s"col1$ix", ix.toByte)
}))
val myRowsSchema = myRows.where("1=1") // "Line 127"
val TempTabName = "MyTempTab"
myRowsSchema.registerTempTable(TempTabName)
The above does not compile:
Error:(127, 31) value where is not a member of
org.apache.spark.rdd.RDD[MyTable]
val myRowsSchema = myRows.where("1=1")
^
However copying the above code into the Spark-Shell, it works - notice we
get a Logical/PhysicalPlan from the heretefore "line 127":
scala> val myRowsSchema = myRows.where("1=1")
myRowsSchema: org.apache.spark.sql.SchemaRDD =
SchemaRDD[29] at RDD at SchemaRDD.scala:102
== Query Plan ==
== Physical Plan ==
Filter 1=1
ExistingRdd [col1#8,col2#9], MapPartitionsRDD[27] at mapPartitions at
basicOperators.scala:219
So .. what is the magic formula for setting up the imports for the
SchemaRDD imports to work properly?
2014-10-02 2:00 GMT-07:00 Stephen Boesch <[email protected]>:
>
> I am noticing disparities in behavior between the REPL and in my
> standalone program in terms of implicit conversion of an RDD to SchemaRDD.
>
> In the REPL the following sequence works:
>
>
> import sqlContext._
>
> val mySchemaRDD = myNormalRDD.where("1=1")
>
>
> However when attempting similar in a standalone program it does not
> compile -with message:
>
> "value where is not a member of org.apache.spark.rdd.RDD[MyRecord]'
>
>
> What is the required recipe for proper implict conversion - given I have
> done the import sqlContext._ in the standalone program as well but it is
> not sufficient there. Note: intellij IDE *does *seem to think that
> "import sqlContext._" were enough - it understands the implicit use of
> "where". But even in IJ it does not actually compile. Rather strange.
>