Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/9232#discussion_r43240828
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -142,6 +145,99 @@ class YarnSparkHadoopUtil extends SparkHadoopUtil {
val containerIdString =
System.getenv(ApplicationConstants.Environment.CONTAINER_ID.name())
ConverterUtils.toContainerId(containerIdString)
}
+
+ /**
+ * Obtains token for the Hive metastore, using the current user as the
principal.
+ * Some exceptions are caught and downgraded to a log message.
+ * @param conf hadoop configuration; the Hive configuration will be
based on this
+ * @return a token, or `None` if there's no need for a token (no
metastore URI or principal
+ * in the config), or if a binding exception was caught and
downgraded.
+ */
+ def obtainTokenForHiveMetastore(conf: Configuration):
Option[Token[DelegationTokenIdentifier]] = {
+ try {
+ obtainTokenForHiveMetastoreInner(conf,
UserGroupInformation.getCurrentUser().getUserName)
+ } catch {
+ case e: Exception => {
+ handleTokenIntrospectionFailure("Hive", e)
+ None
+ }
+ }
+ }
+
+ /**
+ * Handle failures to obtain a token through introspection. Failures to
load the class are
+ * not treated as errors: anything else is.
+ * @param service service name for error messages
+ * @param thrown exception caught
+ * @throws Exception if the `thrown` exception isn't one that is to be
ignored
+ */
+ private[yarn] def handleTokenIntrospectionFailure(service: String,
thrown: Throwable): Unit = {
+ thrown match {
+ case e: ClassNotFoundException =>
+ logInfo(s"$service class not found $e")
+ logDebug("Hive Class not found", e)
+ case t: Throwable => {
--- End diff --
the reason I'd pulled it out was to have an isolated policy which could be
both tested without mocking failures, and be re-used in the HBase token
retrieval, which needs an identical set of clauses.
Given the policy has now been simplified so much, the method is now pretty
much unused; I can pull it. But still the HBase token logic will need to be
100% in sync.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
