Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9875#discussion_r66359015
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -317,14 +318,35 @@ private[spark] class Client(
// and add them as local resources to the application master.
val fs = FileSystem.get(hadoopConf)
val dst = new Path(fs.getHomeDirectory(), appStagingDir)
- val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) + dst
- YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf,
credentials)
// Used to keep track of URIs added to the distributed cache. If the
same URI is added
// multiple times, YARN will fail to launch containers for the app
with an internal
// error.
val distributedUris = new HashSet[String]
- obtainTokenForHiveMetastore(sparkConf, hadoopConf, credentials)
- obtainTokenForHBase(sparkConf, hadoopConf, credentials)
+
+
+ def updateTokens(credentials: Credentials): Unit = {
+ val nns = YarnSparkHadoopUtil.get.getNameNodesToAccess(sparkConf) +
dst
+ YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf,
credentials)
+
+ obtainTokenForHiveMetastore(sparkConf, hadoopConf, credentials)
+ obtainTokenForHBase(sparkConf, hadoopConf, credentials)
+ }
+
+ // If this JVM hosted an yarn-client mode driver before, the
credentials of the current user
+ // now has delegation tokens, which means Hadoop security code will
look at that and not the
+ // keytab login. So we must re-login and get new tokens.
+ if (reusedJVM && loginFromKeytab && !isClusterMode) {
--- End diff --
Hey @harishreedharan , do you plan on updating this patch?
If yes, I'm wondering why not do this in all cases, not just when a new
context is created. The same code should work in both scenarios, right?
If not, should probably close the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
