Github user chesterxgchen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r42811237
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -337,5 +339,82 @@ object YarnSparkHadoopUtil {
conf.getInt("spark.executor.instances", targetNumExecutors)
}
}
+
+ /**
+ * Obtains token for the Hive metastore. Exceptions are caught and logged
+ */
+ def obtainTokenForHiveMetastore(
+ sparkConf: SparkConf,
+ conf: Configuration): Option[Token[DelegationTokenIdentifier]] = {
+ try {
+ obtainTokenForHiveMetastoreInner(sparkConf, conf,
+ UserGroupInformation.getCurrentUser().getUserName)
+ } catch {
+ case e: java.lang.NoSuchMethodException =>
+ logInfo("Hive Method not found " + e)
+ None
+ case e: java.lang.ClassNotFoundException =>
+ logInfo("Hive Class not found " + e)
+ None
+ case e: java.lang.NoClassDefFoundError =>
+ logDebug("Hive Class not found: " + e)
+ None
+ case e: java.lang.ReflectiveOperationException =>
+ // serious problem: log at error and continue
+ logError("Hive Class operation failed: " + e)
+ None
+ case e: RuntimeException =>
+ // any runtime exception, including Illegal Argument Exception
+ throw e
+ case e: Exception => {
+ logError("Unexpected Exception " + e)
+ throw new RuntimeException("Unexpected exception", e)
+ }
+ }
+ }
+
+ /**
+ * Inner routine to obtains token for the Hive metastore; exceptions are
raised on any problem.
+ */
+ private[yarn] def obtainTokenForHiveMetastoreInner(
+ sparkConf: SparkConf,
+ conf: Configuration,
+ username: String): Option[Token[DelegationTokenIdentifier]] = {
+ val mirror = universe.runtimeMirror(getClass.getClassLoader)
+
+ // the hive configuration class is a subclass of Hadoop Configuration
--- End diff --
the hive metadata uri should also in Hadoop Configuration, does it make
sense directly check the metastore URI from conf instead trying to get it from
HiveConf ? which could potentially get exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
