This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6b0c5f8d02de [SPARK-38954][CORE] Support delegation token renewal and
distribution without Kerberos
6b0c5f8d02de is described below
commit 6b0c5f8d02dec66257cbfbb9287a78bf8f7e783e
Author: Parth Chandra <[email protected]>
AuthorDate: Wed Jul 29 17:52:47 2026 -0700
[SPARK-38954][CORE] Support delegation token renewal and distribution
without Kerberos
### What changes were proposed in this pull request?
This PR enables Spark's existing credential distribution mechanism
(driver obtains tokens, pushes them to executors, refreshes before expiry) to
work without Kerberos. Today this mechanism is unconditionally gated
on Kerberos being enabled. This change adds a single configuration switch
(`spark.security.credentials.directProviders.enabled`) that lifts the Kerberos
gate on token collection.
Changes:
- `HadoopDelegationTokenManager.renewalEnabled` now also returns true
when the new config is enabled.
- `obtainDelegationTokens(creds)` and `obtainTokensAndScheduleRenewal()`
branch on whether Kerberos credentials are present: if yes, call providers
inside `doLogin()`/`doAs()` as before; if not (but the switch is
on), call providers directly. Providers that require Kerberos self-gate
via `delegationTokensRequired` returning `false`.
- Per-provider try/catch for failure isolation — one provider throwing
does not block others.
- `SupportsDelegationToken` gains an overridable `tokenManagerRequired()`
hook so scheduler backends can activate the token manager based on the new
config.
- Hard startup failure if `spark.network.crypto.enabled` is not true when
the switch is active (prevents bearer tokens over plaintext RPC).
- WARN log when the config is set but no providers are discovered via
ServiceLoader.
- Note: Standalone clusters are not supported (followup
https://issues.apache.org/jira/browse/SPARK-58329)
Design document: [SPIP: Cloud Credential Refresh and Distribution Without
Kerberos](https://issues.apache.org/jira/browse/SPARK-38954)
### Why are the changes needed?
In non-Kerberos deployments (cloud, internal IdP, cross-account access on
YARN), each executor independently authenticates against identity services,
causing thundering-herd load amplification, inconsistent
credential state, and no centralized control over credential rotation.
The existing distribution infrastructure is already provider-agnostic (Kafka
proves this) — only the activation gates prevent it from running
without Kerberos.
### Does this PR introduce _any_ user-facing change?
Yes. New configuration:
| Config | Default | Description |
|--------|---------|-------------|
| `spark.security.credentials.directProviders.enabled` | `false` |
Enables delegation token collection and renewal without Kerberos. When true,
the manager starts even if Hadoop security is not enabled, calling all
providers whose `delegationTokensRequired` returns true. Requires
`spark.network.crypto.enabled=true`. |
Existing deployments are unaffected (config defaults to false). When
Kerberos IS present, all code paths are unchanged.
### How was this patch tested?
New unit test suite `NonKerberosCredentialsSuite` (7 tests):
- `renewalEnabled` returns true/false based on config
- Providers are called without Kerberos when config is enabled
- Providers with `delegationTokensRequired=false` are skipped
- A failing provider does not prevent other providers from running
- Individual provider can be disabled via
`spark.security.credentials.<service>.enabled`
- Startup fails if `spark.network.crypto.enabled` is not true
Existing `HadoopDelegationTokenManagerSuite` (4 tests) passes unmodified
— verifies no regression in Kerberos deployments.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Co-authored by Claude Code (opus 4.6)
Closes #57285 from parthchandra/cloud-credentials.
Authored-by: Parth Chandra <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../security/HadoopDelegationTokenManager.scala | 140 +++++++++---
.../org/apache/spark/internal/config/package.scala | 12 +
.../spark/scheduler/SupportsDelegationToken.scala | 10 +-
.../cluster/CoarseGrainedSchedulerBackend.scala | 9 +
.../scheduler/local/LocalSchedulerBackend.scala | 4 +
...he.spark.security.HadoopDelegationTokenProvider | 4 +
.../security/NonKerberosCredentialsSuite.scala | 254 +++++++++++++++++++++
docs/security.md | 17 ++
8 files changed, 411 insertions(+), 39 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala
b/core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala
index 58bbd7477bd0..ca202bd9bbfb 100644
---
a/core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala
@@ -34,6 +34,7 @@ import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys
import org.apache.spark.internal.config._
+import org.apache.spark.internal.config.Network.NETWORK_CRYPTO_ENABLED
import org.apache.spark.rpc.RpcEndpointRef
import
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens
import org.apache.spark.security.HadoopDelegationTokenProvider
@@ -76,28 +77,53 @@ private[spark] class HadoopDelegationTokenManager(
require((principal == null) == (keytab == null),
"Both principal and keytab must be defined, or neither.")
+ if (sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)) {
+ val hasEncryption = sparkConf.getBoolean("spark.ssl.rpc.enabled", false) ||
+ (sparkConf.get(NETWORK_AUTH_ENABLED) &&
+ (sparkConf.get(NETWORK_CRYPTO_ENABLED) ||
sparkConf.get(SASL_ENCRYPTION_ENABLED)))
+ require(hasEncryption,
+ "RPC channel encryption must be enabled when " +
+ "spark.security.directCredentialProviders.enabled is true: either " +
+ "spark.ssl.rpc.enabled=true, or spark.authenticate=true together with " +
+ "spark.network.crypto.enabled or
spark.authenticate.enableSaslEncryption. " +
+ "Credential tokens must not be transmitted over unencrypted channels.")
+ }
+
private val delegationTokenProviders = loadProviders()
+ if (sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED) &&
delegationTokenProviders.isEmpty) {
+ logWarning("spark.security.directCredentialProviders.enabled is true but "
+
+ "no HadoopDelegationTokenProvider implementations were discovered. " +
+ "Ensure provider JARs are on the classpath with META-INF/services
registration.")
+ }
logDebug("Using the following builtin delegation token providers: " +
s"${delegationTokenProviders.keys.mkString(", ")}.")
private var renewalExecutor: ScheduledExecutorService = _
+ private def hasKerberosCredentials: Boolean =
+ sparkConf.get(KERBEROS_RENEWAL_CREDENTIALS) match {
+ case "keytab" => principal != null
+ case "ccache" =>
UserGroupInformation.getCurrentUser().hasKerberosCredentials()
+ case _ => false
+ }
+
/** @return Whether delegation token renewal is enabled. */
- def renewalEnabled: Boolean = sparkConf.get(KERBEROS_RENEWAL_CREDENTIALS)
match {
- case "keytab" => principal != null
- case "ccache" =>
UserGroupInformation.getCurrentUser().hasKerberosCredentials()
- case _ => false
+ def renewalEnabled: Boolean = {
+ hasKerberosCredentials ||
+ (sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED) &&
+
delegationTokenProviders.values.exists(_.delegationTokensRequired(sparkConf,
hadoopConf)))
}
/**
- * Start the token renewer. Requires a principal and keytab. Upon start, the
renewer will
- * obtain delegation tokens for all configured services and send them to the
driver, and
- * set up tasks to periodically get fresh tokens as needed.
+ * Start the token renewer. Upon start, the renewer will obtain delegation
tokens for all
+ * configured services and send them to the driver, and set up tasks to
periodically get
+ * fresh tokens as needed.
*
- * This method requires that a keytab has been provided to Spark, and will
try to keep the
- * logged in user's TGT valid while this manager is active.
+ * When Kerberos credentials are available, the manager keeps the TGT
renewed and calls
+ * providers inside a privileged context. When only direct providers are
configured, providers
+ * are called without Kerberos authentication.
*
- * @return New set of delegation tokens created for the configured principal.
+ * @return New set of delegation tokens created for the configured services.
*/
def start(): Array[Byte] = {
require(renewalEnabled, "Token renewal must be enabled to start the
renewer.")
@@ -141,39 +167,58 @@ private[spark] class HadoopDelegationTokenManager(
val hasKerberosCreds = principal != null ||
Option(currentUser.getRealUser()).getOrElse(currentUser).hasKerberosCredentials()
- // Delegation tokens can only be obtained if the real user has Kerberos
credentials, so
- // skip creation when those are not available.
if (hasKerberosCreds) {
val freshUGI = doLogin()
freshUGI.doAs(new PrivilegedExceptionAction[Unit]() {
override def run(): Unit = {
- val (newTokens, _) = obtainDelegationTokens()
+ val (newTokens, _, _) = obtainDelegationTokens()
creds.addAll(newTokens)
}
})
if (!currentUser.equals(freshUGI)) {
FileSystem.closeAllForUGI(freshUGI)
}
+ } else if (sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)) {
+ val (newTokens, _, _) = obtainDelegationTokens(isolateFailures = true)
+ creds.addAll(newTokens)
}
}
/**
* Fetch new delegation tokens for configured services.
*
- * @return 2-tuple (credentials with new tokens, time by which the tokens
must be renewed)
+ * @param isolateFailures When true, catch per-provider exceptions and
continue with remaining
+ * providers. When false, exceptions propagate to the
caller.
+ * @return 3-tuple (credentials with new tokens, time by which the tokens
must be renewed,
+ * number of providers that failed). The failure count is always 0
when
+ * isolateFailures is false (failures propagate instead).
*/
- private def obtainDelegationTokens(): (Credentials, Long) = {
+ private def obtainDelegationTokens(
+ isolateFailures: Boolean = false): (Credentials, Long, Int) = {
val creds = new Credentials()
+ var failureCount = 0
val nextRenewal = delegationTokenProviders.values.flatMap { provider =>
if (provider.delegationTokensRequired(sparkConf, hadoopConf)) {
- provider.obtainDelegationTokens(hadoopConf, sparkConf, creds)
+ if (isolateFailures) {
+ try {
+ provider.obtainDelegationTokens(hadoopConf, sparkConf, creds)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to obtain credentials from " +
+ log"${MDC(LogKeys.SERVICE_NAME, provider.serviceName)}.", e)
+ failureCount += 1
+ None
+ }
+ } else {
+ provider.obtainDelegationTokens(hadoopConf, sparkConf, creds)
+ }
} else {
logDebug(s"Service ${provider.serviceName} does not require a token." +
s" Check your configuration to see if security is disabled or not.")
None
}
}.foldLeft(Long.MaxValue)(math.min)
- (creds, nextRenewal)
+ (creds, nextRenewal, failureCount)
}
// Visible for testing.
@@ -199,8 +244,7 @@ private[spark] class HadoopDelegationTokenManager(
*/
private def updateTokensTask(): Array[Byte] = {
try {
- val freshUGI = doLogin()
- val creds = obtainTokensAndScheduleRenewal(freshUGI)
+ val creds = obtainTokensAndScheduleRenewal()
val tokens = SparkHadoopUtil.get.serialize(creds)
logInfo("Updating delegation tokens.")
@@ -226,24 +270,46 @@ private[spark] class HadoopDelegationTokenManager(
*
* @return Credentials containing the new tokens.
*/
- private def obtainTokensAndScheduleRenewal(ugi: UserGroupInformation):
Credentials = {
- ugi.doAs(new PrivilegedExceptionAction[Credentials]() {
- override def run(): Credentials = {
- val (creds, nextRenewal) = obtainDelegationTokens()
-
- // Calculate the time when new credentials should be created, based on
the configured
- // ratio.
- val now = System.currentTimeMillis
- val ratio = sparkConf.get(CREDENTIALS_RENEWAL_INTERVAL_RATIO)
- val delay = (ratio * (nextRenewal - now)).toLong
- logInfo(log"Calculated delay on renewal is ${MDC(LogKeys.DELAY,
delay)}," +
- log" based on next renewal ${MDC(LogKeys.NEXT_RENEWAL_TIME,
nextRenewal)}" +
- log" and the ratio ${MDC(LogKeys.CREDENTIALS_RENEWAL_INTERVAL_RATIO,
ratio)}," +
- log" and current time ${MDC(LogKeys.CURRENT_TIME, now)}")
- scheduleRenewal(delay)
- creds
- }
- })
+ private def obtainTokensAndScheduleRenewal(): Credentials = {
+ val (creds, nextRenewal, failureCount) = if (hasKerberosCredentials) {
+ val freshUGI = doLogin()
+ freshUGI.doAs(
+ new PrivilegedExceptionAction[(Credentials, Long, Int)]() {
+ override def run(): (Credentials, Long, Int) = {
+ obtainDelegationTokens()
+ }
+ })
+ } else if (sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)) {
+ obtainDelegationTokens(isolateFailures = true)
+ } else {
+ // Reachable when ccache-based Kerberos credentials expired after
start().
+ // Throw so that updateTokensTask() reschedules a retry instead of
silently
+ // distributing empty credentials and never renewing again.
+ throw new IllegalStateException("Cannot obtain delegation tokens: no
Kerberos " +
+ "credentials are available and direct credential providers are not
enabled.")
+ }
+
+ // If at least one provider failed and no credentials were obtained at
all, throw so that
+ // updateTokensTask() skips distributing empty credentials and schedules a
retry. This
+ // mirrors the Kerberos path, where a provider failure propagates instead
of sending empty
+ // tokens to executors. The check keys off the actual credential content
rather than
+ // nextRenewal, since a provider may legitimately add tokens and still
report no expiry
+ // (nextRenewal == Long.MaxValue), which is not a failure.
+ if (failureCount > 0 && creds.numberOfTokens() == 0 &&
creds.numberOfSecretKeys() == 0) {
+ throw new IllegalStateException(
+ "All direct credential providers failed to obtain delegation tokens.")
+ }
+
+ // Scheduling does not require a privileged context (only token
acquisition does).
+ val now = System.currentTimeMillis
+ val ratio = sparkConf.get(CREDENTIALS_RENEWAL_INTERVAL_RATIO)
+ val delay = (ratio * (nextRenewal - now)).toLong
+ logInfo(log"Calculated delay on renewal is ${MDC(LogKeys.DELAY, delay)}," +
+ log" based on next renewal ${MDC(LogKeys.NEXT_RENEWAL_TIME,
nextRenewal)}" +
+ log" and the ratio ${MDC(LogKeys.CREDENTIALS_RENEWAL_INTERVAL_RATIO,
ratio)}," +
+ log" and current time ${MDC(LogKeys.CURRENT_TIME, now)}")
+ scheduleRenewal(delay)
+ creds
}
private def doLogin(): UserGroupInformation = {
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 02e0fc9f0089..8038307afc65 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1730,6 +1730,18 @@ package object config {
.checkValue(_ > 0, "The minimum renewal interval must be a positive time
value.")
.createWithDefaultString("30s")
+ private[spark] val DIRECT_CREDENTIAL_PROVIDERS_ENABLED =
+ ConfigBuilder("spark.security.directCredentialProviders.enabled")
+ .doc(
+ "When true, enables delegation token collection and renewal without
Kerberos. " +
+ "Providers are called directly (without doLogin/doAs) and participate
in the " +
+ "same renewal and distribution lifecycle as Kerberos delegation token
providers. " +
+ "Providers that require Kerberos self-gate via
delegationTokensRequired.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
private[spark] val SHUFFLE_SORT_INIT_BUFFER_SIZE =
ConfigBuilder("spark.shuffle.sort.initialBufferSize")
.internal()
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/SupportsDelegationToken.scala
b/core/src/main/scala/org/apache/spark/scheduler/SupportsDelegationToken.scala
index cbc32f900abf..3fdbe4c25017 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/SupportsDelegationToken.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/SupportsDelegationToken.scala
@@ -33,7 +33,7 @@ private[spark] trait SupportsDelegationToken {
/**
* Create the delegation token manager to be used for the application. This
method is called
* once during the start of the scheduler backend (so after the object has
already been
- * fully constructed), only if security is enabled in the Hadoop
configuration.
+ * fully constructed), when security is enabled or direct credential
providers are configured.
*/
protected def createTokenManager(): Option[HadoopDelegationTokenManager] =
None
@@ -42,8 +42,14 @@ private[spark] trait SupportsDelegationToken {
*/
protected def updateDelegationTokens(tokens: Array[Byte]): Unit
+ /**
+ * Whether the token manager should be started. Returns true if Hadoop
security is enabled
+ * or if direct credential providers are configured.
+ */
+ protected def tokenManagerRequired(): Boolean =
UserGroupInformation.isSecurityEnabled
+
protected def setupTokenManager(): Unit = {
- if (UserGroupInformation.isSecurityEnabled) {
+ if (tokenManagerRequired()) {
delegationTokenManager = createTokenManager()
delegationTokenManager.foreach { dtm =>
val ugi = UserGroupInformation.getCurrentUser
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
index e4bdcd9fa7a3..99fdc534a6fb 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
@@ -618,6 +618,15 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
override def start(): Unit = {
setupTokenManager()
+ if (conf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED) &&
delegationTokenManager.isEmpty) {
+ logWarning("spark.security.directCredentialProviders.enabled is set but
" +
+ "this cluster manager does not support credential distribution. " +
+ "No tokens will be collected or renewed.")
+ }
+ }
+
+ override protected def tokenManagerRequired(): Boolean = {
+ super.tokenManagerRequired() ||
conf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)
}
protected def createDriverEndpoint(): DriverEndpoint = new DriverEndpoint()
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
index c3a1a60a0384..b6a90565aaff 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala
@@ -129,6 +129,10 @@ private[spark] class LocalSchedulerBackend(
Some(new HadoopDelegationTokenManager(conf,
scheduler.sc.hadoopConfiguration, localEndpoint))
}
+ override protected def tokenManagerRequired(): Boolean = {
+ super.tokenManagerRequired() ||
conf.get(config.DIRECT_CREDENTIAL_PROVIDERS_ENABLED)
+ }
+
override protected def updateDelegationTokens(tokens: Array[Byte]): Unit = {
SparkHadoopUtil.get.addDelegationTokens(tokens, conf)
}
diff --git
a/core/src/test/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
b/core/src/test/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
index ed3908e95e4c..8621d66256e0 100644
---
a/core/src/test/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
+++
b/core/src/test/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
@@ -16,3 +16,7 @@
#
org.apache.spark.deploy.security.ExceptionThrowingDelegationTokenProvider
+org.apache.spark.deploy.security.TestNonKerberosTokenProvider
+org.apache.spark.deploy.security.TestDisabledProvider
+org.apache.spark.deploy.security.TestFailingProvider
+org.apache.spark.deploy.security.TestNoExpiryProvider
diff --git
a/core/src/test/scala/org/apache/spark/deploy/security/NonKerberosCredentialsSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/security/NonKerberosCredentialsSuite.scala
new file mode 100644
index 000000000000..d6ecc954cfe0
--- /dev/null
+++
b/core/src/test/scala/org/apache/spark/deploy/security/NonKerberosCredentialsSuite.scala
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.security.Credentials
+import org.mockito.ArgumentCaptor
+import org.mockito.ArgumentMatchers.any
+import org.mockito.Mockito.{mock, never, verify}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.config._
+import org.apache.spark.internal.config.Network.NETWORK_CRYPTO_ENABLED
+import org.apache.spark.rpc.RpcEndpointRef
+import
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens
+import org.apache.spark.security.HadoopDelegationTokenProvider
+
+private class TestNonKerberosTokenProvider extends
HadoopDelegationTokenProvider {
+ override def serviceName: String = "test-direct"
+
+ override def delegationTokensRequired(
+ sparkConf: SparkConf, hadoopConf: Configuration): Boolean =
+ sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ sparkConf: SparkConf,
+ creds: Credentials): Option[Long] = {
+ creds.addSecretKey(new Text("test.direct.credential"),
"test-token".getBytes)
+ Some(System.currentTimeMillis() + 3600000L)
+ }
+}
+
+private class TestDisabledProvider extends HadoopDelegationTokenProvider {
+ override def serviceName: String = "test-disabled"
+
+ override def delegationTokensRequired(
+ sparkConf: SparkConf, hadoopConf: Configuration): Boolean = false
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ sparkConf: SparkConf,
+ creds: Credentials): Option[Long] = {
+ // scalastyle:off throwerror
+ throw new AssertionError("Should not be called when
delegationTokensRequired is false")
+ // scalastyle:on throwerror
+ }
+}
+
+private class TestFailingProvider extends HadoopDelegationTokenProvider {
+ override def serviceName: String = "test-failing"
+
+ override def delegationTokensRequired(
+ sparkConf: SparkConf, hadoopConf: Configuration): Boolean =
+ sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ sparkConf: SparkConf,
+ creds: Credentials): Option[Long] = {
+ throw new RuntimeException("Simulated provider failure")
+ }
+}
+
+// Adds a credential but reports no expiry (returns None), mimicking providers
such as
+// HBaseDelegationTokenProvider. This exercises the nextRenewal ==
Long.MaxValue case where
+// credentials were nevertheless obtained.
+private class TestNoExpiryProvider extends HadoopDelegationTokenProvider {
+ override def serviceName: String = "test-noexpiry"
+
+ override def delegationTokensRequired(
+ sparkConf: SparkConf, hadoopConf: Configuration): Boolean =
+ sparkConf.get(DIRECT_CREDENTIAL_PROVIDERS_ENABLED)
+
+ override def obtainDelegationTokens(
+ hadoopConf: Configuration,
+ sparkConf: SparkConf,
+ creds: Credentials): Option[Long] = {
+ creds.addSecretKey(new Text("test.noexpiry.credential"),
"noexpiry-token".getBytes)
+ None
+ }
+}
+
+class NonKerberosCredentialsSuite extends SparkFunSuite {
+ private val hadoopConf = new Configuration()
+
+ private def baseConf: SparkConf = new SparkConf(false)
+ .set(DIRECT_CREDENTIAL_PROVIDERS_ENABLED, true)
+ .set(NETWORK_AUTH_ENABLED, true)
+ .set(NETWORK_CRYPTO_ENABLED, true)
+
+ test("renewalEnabled returns true when config is enabled") {
+ val manager = new HadoopDelegationTokenManager(baseConf, hadoopConf, null)
+ assert(manager.renewalEnabled)
+ }
+
+ test("renewalEnabled returns false when config is disabled and no Kerberos")
{
+ val sparkConf = new SparkConf(false)
+ .set(NETWORK_CRYPTO_ENABLED, true)
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+ assert(!manager.renewalEnabled)
+ }
+
+ test("providers are called without Kerberos when config is enabled") {
+ val manager = new HadoopDelegationTokenManager(baseConf, hadoopConf, null)
+
+ val creds = new Credentials()
+ manager.obtainDelegationTokens(creds)
+
+ assert(creds.getSecretKey(new Text("test.direct.credential")) != null)
+ assert(new String(creds.getSecretKey(new Text("test.direct.credential")))
=== "test-token")
+ }
+
+ test("providers with delegationTokensRequired=false are not called") {
+ val manager = new HadoopDelegationTokenManager(baseConf, hadoopConf, null)
+
+ val creds = new Credentials()
+ manager.obtainDelegationTokens(creds)
+
+ assert(creds.getSecretKey(new Text("test.direct.credential")) != null)
+ }
+
+ test("provider failure does not prevent other providers from running") {
+ val sparkConf = baseConf
+ .set("spark.security.credentials.test-failing.enabled", "true")
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+
+ val creds = new Credentials()
+ manager.obtainDelegationTokens(creds)
+
+ assert(creds.getSecretKey(new Text("test.direct.credential")) != null)
+ assert(new String(creds.getSecretKey(new Text("test.direct.credential")))
=== "test-token")
+ }
+
+ test("individual provider can be disabled via per-service config") {
+ val sparkConf = baseConf
+ .set("spark.security.credentials.test-direct.enabled", "false")
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+
+ assert(!manager.isProviderLoaded("test-direct"))
+ }
+
+ test("fails if no RPC encryption is enabled") {
+ val sparkConf = new SparkConf(false)
+ .set(DIRECT_CREDENTIAL_PROVIDERS_ENABLED, true)
+
+ val e = intercept[IllegalArgumentException] {
+ new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+ }
+ assert(e.getMessage.contains("RPC channel encryption"))
+ }
+
+ test("accepts SASL encryption as sufficient") {
+ val sparkConf = new SparkConf(false)
+ .set(DIRECT_CREDENTIAL_PROVIDERS_ENABLED, true)
+ .set(NETWORK_AUTH_ENABLED, true)
+ .set(SASL_ENCRYPTION_ENABLED, true)
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+ assert(manager.renewalEnabled)
+ }
+
+ test("accepts SSL RPC encryption as sufficient") {
+ val sparkConf = new SparkConf(false)
+ .set(DIRECT_CREDENTIAL_PROVIDERS_ENABLED, true)
+ .set("spark.ssl.rpc.enabled", "true")
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf, null)
+ assert(manager.renewalEnabled)
+ }
+
+ test("start() obtains tokens and sends UpdateDelegationTokens to
schedulerRef") {
+ val mockRef = mock(classOf[RpcEndpointRef])
+ val manager = new HadoopDelegationTokenManager(baseConf, hadoopConf,
mockRef)
+
+ try {
+ val tokens = manager.start()
+ assert(tokens != null)
+
+ val captor = ArgumentCaptor.forClass(classOf[Any])
+ verify(mockRef).send(captor.capture())
+ val msg = captor.getValue.asInstanceOf[UpdateDelegationTokens]
+
+ val creds = SparkHadoopUtil.get.deserialize(msg.tokens)
+ assert(creds.getSecretKey(new Text("test.direct.credential")) != null)
+ assert(new String(creds.getSecretKey(new
Text("test.direct.credential"))) === "test-token")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("start() does not send empty credentials when all direct providers
fail") {
+ val mockRef = mock(classOf[RpcEndpointRef])
+ // Disable both succeeding providers so the failing provider is the sole
active one,
+ // producing a total failure with no credentials obtained.
+ val sparkConf = baseConf
+ .set("spark.security.credentials.test-direct.enabled", "false")
+ .set("spark.security.credentials.test-noexpiry.enabled", "false")
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf,
mockRef)
+
+ try {
+ // On total failure, obtainTokensAndScheduleRenewal throws so
updateTokensTask() skips
+ // distributing empty credentials and schedules a retry instead. start()
returns null
+ // and no UpdateDelegationTokens message is ever sent to the executors.
+ val tokens = manager.start()
+ assert(tokens == null)
+ verify(mockRef, never()).send(any())
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("start() sends partial credentials when some providers succeed without
expiry") {
+ val mockRef = mock(classOf[RpcEndpointRef])
+ // Disable the provider that reports an expiry, leaving a provider that
adds a credential
+ // but returns None (nextRenewal stays Long.MaxValue) alongside the
failing one. Even
+ // though a provider failed and no expiry was reported, the obtained
credential must still
+ // be distributed rather than discarded as a spurious total failure.
+ val sparkConf = baseConf
+ .set("spark.security.credentials.test-direct.enabled", "false")
+ val manager = new HadoopDelegationTokenManager(sparkConf, hadoopConf,
mockRef)
+
+ try {
+ val tokens = manager.start()
+ assert(tokens != null)
+
+ val captor = ArgumentCaptor.forClass(classOf[Any])
+ verify(mockRef).send(captor.capture())
+ val msg = captor.getValue.asInstanceOf[UpdateDelegationTokens]
+
+ val creds = SparkHadoopUtil.get.deserialize(msg.tokens)
+ assert(creds.getSecretKey(new Text("test.noexpiry.credential")) != null)
+ assert(new String(creds.getSecretKey(new
Text("test.noexpiry.credential")))
+ === "noexpiry-token")
+ } finally {
+ manager.stop()
+ }
+ }
+}
diff --git a/docs/security.md b/docs/security.md
index 7c0d1b08875e..d80250256066 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -981,6 +981,23 @@ The following options provides finer-grained control for
this feature:
</td>
<td>2.3.0</td>
</tr>
+<tr>
+ <td><code>spark.security.directCredentialProviders.enabled</code></td>
+ <td><code>false</code></td>
+ <td>
+ When true, enables credential collection and renewal without Kerberos.
Providers
+ registered via <code>HadoopDelegationTokenProvider</code> are called
directly (without
+ <code>doLogin</code>/<code>doAs</code>) and participate in the same
renewal and
+ distribution lifecycle as Kerberos delegation token providers. Providers
that require
+ Kerberos self-gate via their <code>delegationTokensRequired</code> method.
+ Requires RPC channel encryption: either
<code>spark.ssl.rpc.enabled=true</code>,
+ or <code>spark.authenticate=true</code> together with
+ <code>spark.network.crypto.enabled</code> or
+ <code>spark.authenticate.enableSaslEncryption</code>.
+ Supported on YARN, Kubernetes, and local mode. Not supported on standalone
clusters.
+ </td>
+ <td>4.3.0</td>
+</tr>
<tr>
<td><code>spark.kerberos.access.hadoopFileSystems</code></td>
<td>(none)</td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]