Hi,
Even though hive.metastore.warehouse.dir in hive-site.xml is set to the
default user/hive/warehouse and the permissions are correct in hdfs,
HiveContext seems to be creating metastore locally instead of hdfs. After
looking into the spark code, I found the following in HiveContext.scala:
/**
* SQLConf and HiveConf contracts: when the hive session is first
initialized, params in
* HiveConf will get picked up by the SQLConf. Additionally, any properties
set by
* set() or a SET command inside hql() or sql() will be set in the SQLConf
*as well as*
* in the HiveConf.
*/
@transient protected[hive] lazy val hiveconf = new
HiveConf(classOf[SessionState])
@transient protected[hive] lazy val sessionState = {
val ss = new SessionState(hiveconf)
set(hiveconf.getAllProperties) // Have SQLConf pick up the initial set
of HiveConf.
ss
}
It seems as though when a HiveContext is created, it is launched without any
configuration and hive-site.xml is not used to set properties. It looks like
I can set properties after creation by using hql() method but what I am
looking for is for the hive context to be initialized according to the
configuration in hive-site.xml at the time of initialization. Any help would
be greatly appreciated!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/HiveContext-is-creating-metastore-warehouse-locally-instead-of-in-hdfs-tp10838.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
