spark on yarn with hive

[centerqi hu]

I want to let hive run on spark and yarn clusters,Hive Metastore is stored
in MySQL

I compiled spark code:
sh make-distribution.sh --hadoop 2.4.1 --with-yarn --skpi-java-test --tgz
--with-hive

My HQL code:
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql._
import org.apache.spark.sql.hive.LocalHiveContext

object HqlTest {
  case class Record(key: Int, value: String)
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("HiveFromSpark")
    val sc = new SparkContext(sparkConf)

    val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
    import hiveContext._
    hiveContext.hql("FROM tmp_adclick_udc select
uid").collect().foreach(println)
  }
}

I submitted the job code:
/usr/local/webserver/sparkhive/bin/spark-submit --class HqlTest --master
yarn --deploy-mode cluster --queue sls_queue_1 --num-executors 5
--driver-memory 6g --executor-memory 20g --executor-cores 5
target/scala-2.10/simple-project_2.10-1.0.jar
/user/www/udc/output/2014-08-10/*
/user/www/udc/input/platformuvpv2/2014-08-10

I put hive-site.xml into spark/conf/.

But, I get the following error:

14/08/30 16:30:49 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
14/08/30 16:30:49 INFO Datastore: The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
14/08/30 16:30:54 ERROR Hive:
NoSuchObjectException(message:default.tmp_adclick_udc table not found)
        at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1373)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
        at com.sun.proxy.$Proxy26.get_table(Unknown Source)
        at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

thanks
-- 
cente...@gmail.com|齐忠