Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/22624
If we are talking about the token renewal functionality, could we possibly
refactor the `HadoopFSDelegationTokenProvider` as well. I found that within the
function `obtainDelegationTokens()`:
This code-block:
```
val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),
fsToGetTokens, creds)
// Get the token renewal interval if it is not set. It will only be
called once.
if (tokenRenewalInterval == null) {
tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf,
fsToGetTokens)
}
```
calls `fetchDelegationTokens()` twice since the `tokenRenewalInterval` will
always be null upon creation of the `TokenManager` which I think is unnecessary
in the case of Kubernetes (as you are creating 2 DTs when only one is needed.)
Idk if use-case is different in Mesos / Yarn, but could this possibly be
refactored to only call `fetchDelegationTokens()` once upon startup or to have
a param to specify `tokenRenewalInterval`? I could send a follow-up PR if
desired, but idk if this fits better within the scope of this PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
