Github user yolandagao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9272#discussion_r43974652
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -150,6 +152,14 @@ private[hive] class ClientWrapper(
val original = Thread.currentThread().getContextClassLoader
// Switch to the initClassLoader.
Thread.currentThread().setContextClassLoader(initClassLoader)
+
+ val sparkConf = new SparkConf
+ if (sparkConf.contains("spark.yarn.principal") &&
sparkConf.contains("spark.yarn.keytab")) {
+ UserGroupInformation.loginUserFromKeytab(
+ sparkConf.get("spark.yarn.principal"),
--- End diff --
@steveloughran Good point. Better to check the existence of the keytab file
before make the login call, as if the keytab doesn't exist the UGI call will
definitely fail but with some indirect message like "login failed... no keys
found..." ect. Added the check.
However, calling SparkHadoopUtil.get.loginUserFromKeytab instead of
UserGroupInformation.loginUserFromKeytab in ClientWrapper will not solve the
problem as SparkHadoopUtil is shared and the UserGroupInformation class it
includes is not the same one used by SessionState.start in ClientWrapper.
Therefore, the program still fails with no tgt exception when connecting to
metastore. Also not able to replace the UGI call in SparkSubmit either, as
incorrect type of SparkHadoopUtil instance might get created due to yarn mode
isn't set in the system until it flows to yarn Client.scala.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
