Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22704#discussion_r226069637
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
---
@@ -49,8 +49,11 @@ private[deploy] class
HadoopFSDelegationTokenProvider(fileSystems: Configuration
val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),
fsToGetTokens, creds)
// Get the token renewal interval if it is not set. It will only be
called once.
- if (tokenRenewalInterval == null) {
- tokenRenewalInterval = getTokenRenewalInterval(hadoopConf,
sparkConf, fsToGetTokens)
+ // If running a Kerberos job on Kubernetes, you may specify that you
wish to not
+ // obtain the tokenRenewal interval, as the renewal service may be
external.
--- End diff --
What does "as the renewal service may be external" mean?
If you're in this code, there are two options:
- you want Spark to renew tokens, in which case you need the interval.
- you do not want Spark to renew tokens, in which case you should not give
Spark neither a principal and a keytab.
The principal/keytab combo is NOT a replacement for kinit. It has always
been, and always will be, the way to tell Spark that you want Spark to renew
tokens itself. The current k8s backend is broken in that regard.
And BTW, I know what you mean when you mention an external renewal service.
But again, that does not exist, and until it does, you should not do things
that assume its existence.
Now as for how to avoid the extra token, that's does not need a
configuration at all. The extra token is needed in YARN because to know the
renewal interval, you have to call `renew()` on the token, and that fails with
the token created with YARN as the renewer.
So to fix this:
- check that this is running in YARN, and create the extra token
- if it's not running on YARN, just call "renew()" on the existing token
And when, and if, there is an external renewal service, a lot of this will
have to change in the first place.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
