[
https://issues.apache.org/jira/browse/SPARK-15830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yesha Vora updated SPARK-15830:
-------------------------------
Affects Version/s: 1.6.1
> Spark application should get hive tokens only when it is required
> -----------------------------------------------------------------
>
> Key: SPARK-15830
> URL: https://issues.apache.org/jira/browse/SPARK-15830
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: Yesha Vora
>
> Currently , All spark application try to get Hive tokens (Even if application
> does not use them) if Hive is installed on the cluster.
> Due to this practice, spark application which does not require Hive fails
> when Hive service (metastore) is down for some reason.
> Thus, spark should only try to get Hive tokens when required. It should not
> fetch hive token if it is not needed by application.
> Example : Spark Pi application does not perform any hive related actions. But
> Spark Pi application still fails if hive metastore service is down.
> {code}
> 16/06/08 01:18:42 INFO YarnSparkHadoopUtil: getting token for namenode:
> hdfs://xxx:8020/user/xx/.sparkStaging/application_1465347287950_0001
> 16/06/08 01:18:42 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 7 for
> xx on xx.xx.xx.xxx:8020
> 16/06/08 01:18:43 INFO metastore: Trying to connect to metastore with URI
> thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:43 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:43 INFO metastore: Waiting 5 seconds before next connection
> attempt.
> 16/06/08 01:18:48 INFO metastore: Trying to connect to metastore with URI
> thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:48 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:48 INFO metastore: Waiting 5 seconds before next connection
> attempt.
> 16/06/08 01:18:53 INFO metastore: Trying to connect to metastore with URI
> thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:53 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:53 INFO metastore: Waiting 5 seconds before next connection
> attempt.
> 16/06/08 01:18:59 WARN Hive: Failed to access metastore. This class should
> not accessed in runtime.
> org.apache.hadoop.hive.ql.metadata.Hive Exception : java.lang.Runtime
> Exception : Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
> at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
