Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22704#discussion_r226391491
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
---
@@ -49,8 +49,11 @@ private[deploy] class
HadoopFSDelegationTokenProvider(fileSystems: Configuration
val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),
fsToGetTokens, creds)
// Get the token renewal interval if it is not set. It will only be
called once.
- if (tokenRenewalInterval == null) {
- tokenRenewalInterval = getTokenRenewalInterval(hadoopConf,
sparkConf, fsToGetTokens)
+ // If running a Kerberos job on Kubernetes, you may specify that you
wish to not
+ // obtain the tokenRenewal interval, as the renewal service may be
external.
--- End diff --
This is going to be kind of a long reply. I've explained most of this in a
long comment I left in your k8s-krb google doc. I should probably put that in a
readme here since it's really easy to get confused.
> an external token service that places tokens into Secrets may exist
within a company organization
Lots of things here.
1. You're talking about a service that creates tokens and gives them to
Spark. That is unrelated to the code you're touching; this code is Spark's code
for creating its own tokens. In the service scenario you're talking about, this
code either wouldn't run at all, or would be a no-op. (And if that doesn't
happen it's a bug somewhere in Spark. That's also, e.g., how Oozie gives Spark
delegation tokens for other services.)
2. As I mention above, this code is about Spark creating its own tokens. If
you're talking about a "renewal service" in this context, you need to define
the interface between Spark and the service, so Spark can authenticate and
provide the tokens it created to that service.
YARN has that functionality, but it's all very specific to YARN, and the
code lives in the YARN backend in Spark, not here. If k8s had that
functionality natively, you could do that in the k8s backend. But if you talk
about a generic external service without any code in Spark to interact with
that service, that's what I keep saying doesn't exist.
3. The renewal you talk about and the renewal this code talks about are not
the same thing.
Your renewal is the standard delegation token renewal: you create it, and
renew it periodically until it reaches its lifetime. In the YARN case, the YARN
RM does that renewal for you, so your tokens will remain valid up to when the
application finishes or the token expires, whatever happens first.
The renewal this code talks about is not that. It's meant to work around
the upper bound described above. Without it, your app has a maximum lifetime
which is the lifetime of the token. This code gives Spark the ability to create
new tokens and distribute them to new and already running executors, so the app
keeps working indefinitely.
In this mode, Spark doesn't "renew" tokens in the sense you talk about, it
just creates new ones once it's getting close to the renewal time.
But for that to work you need a Kerberos login. Thus, you need a principal
and a keytab, unless you want to login to the driver's host and kinit every
time the TGT needs renewal.
On the other hand, there is no reason why this code couldn't try to do what
you mean by renewal. But I'm pretty sure that renewal still requires a valid
kerberos login. And just like delegation tokens, TGTs expire. The service
talked about in (1) above could help, but this is the only current solution
(bar YARN) where Spark creates its own tokens, and also allows the application
to run past the token's expiration.
There are also other caveats, like certain DTs (like Hive's) not being
renewable.
> The design was specifically for the popular use-case in which for
cluster-mode we would not send keytabs around and instead read the DT from a
secret.
See above what the scenario that requires keytabs is about. It's not great
to distribute keytabs, but that's the only current solution to that problem.
> cases where the Driver is over-worked in running this renewal service
This code runs once a day (in the case of usual service configs) and just
performs a few RPCs to get new tokens and send them to executors. It's hardly a
heavy weight thing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
