Hi,
In Spark 2 I am using sbt or mvn to compile my scala program. This used to
compile and run perfectly with Spark 1.6.1 but now it is throwing error
I believe the problem is here. I have
name := "scala"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
However the error I am getting is
[error] bad symbolic reference. A signature in HiveContext.class refers to
type Logging
[error] in package org.apache.spark which is not available.
[error] It may be completely missing from the current classpath, or the
version on
[error] the classpath might be incompatible with the version used when
compiling HiveContext.class.
[error] one error found
[error] (compile:compileIncremental) Compilation failed
And this is the code
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.types._
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
object ETL_scratchpad_dummy {
def main(args: Array[String]) {
val conf = new SparkConf().
setAppName("ETL_scratchpad_dummy").
set("spark.driver.allowMultipleContexts", "true").
set("enableHiveSupport","true")
val sc = new SparkContext(conf)
//import sqlContext.implicits._
val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
HiveContext.sql("use oraclehadoop")
Anyone has come across this
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
