Hi all,
I have several Hive queries that work in spark-shell, but they don't work in
spark-submit. In fact, I can't even show all databases. The following works
in spark-shell:
import org.apache.spark._
import org.apache.spark.sql._
object ViewabilityFetchInsertDailyHive {
def main() {
val x = sqlContext.sql("show databases")
val z = x.collect
for(i <- z) println(i.toString)
}
}
But the following doesn't work in spark-submit:
object PrintAllDatabases {
def main() {
val sc = new SparkContext(new
SparkConf().setAppName(this.getClass.getName))
val sqlContext = new SQLContext(sc)
val x = sqlContext.sql("show databases")
val z = x.collect
for(i <- z) println(i.toString)
}
}
And I get this error:
16/03/14 22:27:55 INFO BlockManagerMaster: Registered BlockManager
16/03/14 22:27:56 INFO EventLoggingListener: Logging events to
hdfs://nameservice1/user/spark/applicationHistory/local-1457994475020
Exception in thread "main" java.lang.RuntimeException: [1.1] failure:
``with'' expected but identifier show found
show databases
^
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
at
org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
at
org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
at
org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
at
org.apache.spark.sql.execution.SparkSQLParser$$anonfun$org$apache$spark$sql$execution$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:114)
Any suggestions are appreciated!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Hive-query-works-in-spark-shell-not-spark-submit-tp26492.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
