Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/22704#discussion_r226127311
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
---
@@ -49,8 +49,11 @@ private[deploy] class
HadoopFSDelegationTokenProvider(fileSystems: Configuration
val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),
fsToGetTokens, creds)
// Get the token renewal interval if it is not set. It will only be
called once.
- if (tokenRenewalInterval == null) {
- tokenRenewalInterval = getTokenRenewalInterval(hadoopConf,
sparkConf, fsToGetTokens)
+ // If running a Kerberos job on Kubernetes, you may specify that you
wish to not
+ // obtain the tokenRenewal interval, as the renewal service may be
external.
--- End diff --
> And BTW, I know what you mean when you mention an external renewal
service. But again, that does not exist, and until it does, you should not do
things that assume its existence.
Right, so in Spark there is no token renewal service ATM, but the existence
of an external token service that places tokens into Secrets may exist within a
company organization, no? Whether they leverage one, provided by Spark or not.
So I personally thought that such a comment is sensible. But I'll remove it
based on your reasoning.
> If you're in this code, there are two options:
Ah, very good point, checking for the presence of a keytab / principal as
a flag, given that
> It has always been, and always will be, the way to tell Spark that you
want Spark to renew tokens itself
makes sense
> The current k8s backend is broken in that regard.
The design was specifically for the popular use-case in which for
cluster-mode we would not send keytabs around and instead read the DT from a
secret. So true, it does break the traditional design because we are using a
keytab and not enabling renewal. Contractually with your work, next steps would
be to parallel the other resource-managers in allowing for the option to use
the renewal code if the keytab and principal is already in the Driver. Just for
interest, has there been any cases where the Driver is over-worked in running
this renewal service and managing the DTs for many executors? In essence, could
there be any convincing use-case to have the Driver use the keytab for login,
but not want to do its own renewal as it might be the case that it can't handle
the load?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
