This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 057b60796898 [SPARK-57893][CORE] Implement `UserCredentialManager`
057b60796898 is described below
commit 057b60796898749a641c160c706dad4c6885da70
Author: Kousuke Saruta <[email protected]>
AuthorDate: Wed Jul 29 07:37:47 2026 +0900
[SPARK-57893][CORE] Implement `UserCredentialManager`
### What changes were proposed in this pull request?
This is a part of the SPIP: OIDC Credential Propagation (SPARK-57703).
This PR adds `UserCredentialManager`, the driver-side credential renewal
loop.
`UserCredentialManager` is a sibling of `HadoopDelegationTokenManager` and
handles the OIDC credential propagation path independently. Both managers run
on independent threads and can both be active simultaneously.
**Responsibilities:**
1. Reads the current identity token via `TokenIngestor.load()`
2. Calls `CredentialProvider.resolve()` for each configured scheme
3. Serializes the resulting `UserCredentials` and invokes the propagation
callback
4. Schedules renewal based on `min(identity token expiry, service
credential expiry) - safetyMargin`
5. Retries with exponential backoff on failure
**Key design decisions:**
- **Fail-fast on startup:** If the initial credential acquisition fails,
the application fails to start rather than running with null credentials.
- **Per-provider error isolation:** One provider failure does not abort the
entire resolution loop. Other providers continue to resolve.
- **Separation of concerns:** Credential *resolution* failures trigger
backoff; *propagation* failures are logged and retried on the next scheduled
renewal.
- **ObjectInputFilter:** Deserialization is restricted to only the classes
needed for `UserCredentials`, preventing deserialization attacks.
**New files:**
- `core/.../deploy/security/UserCredentialManager.scala`: Main
implementation
- `core/.../deploy/security/UserCredentialManagerSuite.scala`: Unit tests
**Modified files:**
- `CredentialProviderLoader.java`: Added `discoverAllSchemes()` for
auto-discovery of registered schemes
- `config/package.scala`: Added `spark.security.oidc.*` configuration keys
### Why are the changes needed?
There is no component to orchestrate credential acquisition, renewal
scheduling, and propagation triggering on the driver for OIDC-based
environments. This is the core driver-side engine that coordinates
`TokenIngestor` and `CredentialProvider` to maintain fresh credentials.
### Does this PR introduce _any_ user-facing change?
No. All new behavior is gated by `spark.security.oidc.enabled=false`
(default). No existing APIs are changed.
### How was this patch tested?
Unit tests in `UserCredentialManagerSuite` covering:
- Initial credential acquisition and callback invocation
- Fail-fast when TokenIngestor returns empty
- Serialization/deserialization round-trip with ObjectInputFilter security
- `computeRenewalDelay` edge cases (safety margin, min interval, no expiry)
- Exponential backoff (growth, cap, zero-failure edge case)
- Per-provider error isolation (partial failure, total failure)
- Token rotation triggers new credential acquisition and propagation
- Factory method (`create`) with various configurations
- Lifecycle safety (double-start guard, stop after start)
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #57387 from
sarutak/oidc-propagation/subtask4-user-credential-manager.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
(cherry picked from commit 86de29d22894b4ad131c6fbdedf953945e2dae24)
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../spark/security/CredentialProviderLoader.java | 33 +-
.../deploy/security/UserCredentialManager.scala | 448 +++++++++++++++++++
.../org/apache/spark/internal/config/package.scala | 40 ++
.../security/UserCredentialManagerSuite.scala | 478 +++++++++++++++++++++
4 files changed, 998 insertions(+), 1 deletion(-)
diff --git
a/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
b/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
index b22a6aed2b55..9d6e9e6fedf0 100644
--- a/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
+++ b/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
@@ -20,6 +20,7 @@ package org.apache.spark.security;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.IdentityHashMap;
import java.util.List;
import java.util.Locale;
@@ -30,6 +31,8 @@ import java.util.ServiceLoader;
import java.util.Set;
import java.util.stream.Collectors;
+import com.google.common.annotations.VisibleForTesting;
+
import org.apache.spark.annotation.Private;
/**
@@ -221,10 +224,38 @@ public final class CredentialProviderLoader {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances
and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set.
Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or
apply
+ * explicit selection rules -- it only reports what schemes are potentially
available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme
configuration
+ * (e.g., {@code spark.security.oidc.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new HashSet<>();
+ for (CredentialProvider provider : providers) {
+ Set<String> providerSchemes = provider.supportedSchemes();
+ if (providerSchemes != null) {
+ for (String s : providerSchemes) {
+ schemes.add(s.toLowerCase(Locale.ROOT));
+ }
+ }
+ }
+ return schemes;
+ }
+
/**
* Resets the cached provider list and initialization tracking. Intended for
testing only.
*/
- static void resetForTesting() {
+ @VisibleForTesting
+ public static void resetForTesting() {
synchronized (CredentialProviderLoader.class) {
cachedProviders = null;
initializedProviders.clear();
diff --git
a/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
b/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
new file mode 100644
index 000000000000..c98771ee0265
--- /dev/null
+++
b/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream,
ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException,
ScheduledExecutorService, TimeUnit}
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the
OIDC credential
+ * propagation path independently. Both managers run on independent threads
and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3
via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the
propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service
credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.oidc.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_OIDC_RENEWAL_MIN_INTERVAL)
+
+ // Counter for exponential backoff calculation.
+ // Only accessed from the single-thread renewal executor.
+ private var consecutiveFailures: Int = 0
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ private var renewalExecutor: ScheduledExecutorService = _
+
+ // Snapshot of credential-related configuration at construction time.
+ private val credentialConfMap: java.util.Map[String, String] =
sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.oidc."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules
renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be
loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent
renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than
once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be
obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing
or malformed. " +
+ s"Check ${SECURITY_OIDC_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized =
UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+
ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials,
propagate,
+ * and schedule the next renewal. Failures are retried with exponential
backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing
or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized =
UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution
failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to
propagate " +
+ log"to executors. Executors will receive updated credentials on
next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is
intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks
credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next
scheduled renewal.
+ consecutiveFailures = 0
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case NonFatal(e) =>
+ consecutiveFailures += 1
+ val failures = consecutiveFailures
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS,
UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ case t: Throwable =>
+ // A fatal error would otherwise be swallowed by the executor,
silently stopping
+ // all future renewals. Log it before propagating.
+ logError(log"Fatal error during credential renewal; no further
renewals will be " +
+ log"scheduled.", t)
+ throw t
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service
credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme,
credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial
resolution.
+ // The full URI is passed in future versions when path-specific
resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but
a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme,
UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ if (credential == null) {
+ logWarning(log"Provider for scheme ${MDC(LogKeys.URI, scheme)} " +
+ log"returned null; skipping.")
+ } else {
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) =>
Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured
correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.oidc.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all
providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers
may
+ * support the same scheme. In that case,
`CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception
catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme names.
+ // Keys are of the form spark.security.oidc.provider.<scheme> or
+ // spark.security.oidc.provider.<scheme>.<subkey>. We extract only the
first
+ // segment after the prefix to get the scheme name.
+ val providerPrefix = "spark.security.oidc.provider."
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith(providerPrefix) }
+ .map { case (k, _) => k.stripPrefix(providerPrefix).split('.').head }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a
provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws
for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] =
Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or
service
+ // credentials. Use half of the default suggested TTL (15 min / 2 =
7.5 min,
+ // rounded down) as a conservative polling interval. This ensures
credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures
should always
+ // be >= 1 when this is called (incremented before calling), but we
protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently).
This is expected
+ // during application shutdown -- no further renewals will be
scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider
resolution.
+ * Providers should not rely on this value; it signals that no specific
endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between
retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service
credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long =
TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes
needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks
while
+ * allowing Java's built-in collection internal classes and arrays that
HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
+ "org.apache.spark.security.**;" +
+ "java.util.**;" +
+ "java.time.**;" +
+ "java.lang.**;" +
+ "maxdepth=10;" +
+ "maxarray=1000;" +
+ "maxrefs=1000;" +
+ "!*"
+
+ /**
+ * Create a UserCredentialManager if OIDC credential propagation is enabled.
+ *
+ * @param sparkConf The Spark configuration
+ * @param onCredentialsUpdate Callback to propagate credentials to executors
+ * @return Some(manager) if enabled, None otherwise
+ */
+ def create(
+ sparkConf: SparkConf,
+ onCredentialsUpdate: Array[Byte] => Unit): Option[UserCredentialManager]
= {
+ if (!sparkConf.get(SECURITY_OIDC_ENABLED)) {
+ None
+ } else {
+ val tokenFile =
sparkConf.get(SECURITY_OIDC_IDENTITY_TOKEN_FILE).getOrElse {
+ throw new IllegalArgumentException(
+ s"${SECURITY_OIDC_IDENTITY_TOKEN_FILE.key} must be set when " +
+ s"${SECURITY_OIDC_ENABLED.key} is true")
+ }
+
+ val tokenIngestor = new FileTokenIngestor(Paths.get(tokenFile))
+ Some(new UserCredentialManager(sparkConf, tokenIngestor,
onCredentialsUpdate))
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials:
UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ Utils.tryWithResource(new ObjectOutputStream(bos)) { oos =>
+ oos.writeObject(credentials)
+ oos.flush()
+ }
+ bos.toByteArray
+ }
+
+ /**
+ * Deserialize [[UserCredentials]] from a byte array.
+ *
+ * Uses an [[ObjectInputFilter]] to restrict deserialized classes to only
those
+ * required for [[UserCredentials]], preventing deserialization attacks.
+ */
+ def deserializeUserCredentials(bytes: Array[Byte]): UserCredentials = {
+ val bis = new java.io.ByteArrayInputStream(bytes)
+ Utils.tryWithResource(new ObjectInputStream(bis)) { ois =>
+ ois.setObjectInputFilter(
+ ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
+ ois.readObject().asInstanceOf[UserCredentials]
+ }
+ }
+}
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 56e384ab49c6..7131271cac6b 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1690,6 +1690,46 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_OIDC_ENABLED =
+ ConfigBuilder("spark.security.oidc.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the
driver reads an " +
+ "identity token from a file, exchanges it for short-lived service
credentials via " +
+ "CredentialProvider implementations, and propagates those credentials
to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_OIDC_IDENTITY_TOKEN_FILE =
+ ConfigBuilder("spark.security.oidc.identityToken.file")
+ .doc("Path to the OIDC identity token file on the driver. Required when
" +
+ "spark.security.oidc.enabled is true. The file should contain a JWT
token " +
+ "(e.g., a Kubernetes projected service account token).")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .stringConf
+ .createOptional
+
+ private[spark] val SECURITY_OIDC_RENEWAL_SAFETY_MARGIN =
+ ConfigBuilder("spark.security.oidc.renewal.safetyMargin")
+ .doc("How long before credential expiry to trigger renewal. Credentials
are refreshed " +
+ "at min(identity token expiry, service credential expiry) minus this
margin.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .checkValue(_ > 0, "The safety margin must be a positive time value.")
+ .createWithDefaultString("60s")
+
+ private[spark] val SECURITY_OIDC_RENEWAL_MIN_INTERVAL =
+ ConfigBuilder("spark.security.oidc.renewal.minInterval")
+ .doc("Minimum interval between credential renewal attempts. This
prevents tight renewal " +
+ "loops when credentials have very short TTLs or when failures cause
rapid retries.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .checkValue(_ > 0, "The minimum renewal interval must be a positive time
value.")
+ .createWithDefaultString("30s")
+
private[spark] val SHUFFLE_SORT_INIT_BUFFER_SIZE =
ConfigBuilder("spark.shuffle.sort.initialBufferSize")
.internal()
diff --git
a/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
new file mode 100644
index 000000000000..b5ee471013b3
--- /dev/null
+++
b/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.time.Instant
+import java.util
+import java.util.Optional
+import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
+
+import scala.concurrent.duration._
+
+import org.scalatest.concurrent.Eventually.{eventually, timeout}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+
+class UserCredentialManagerSuite extends SparkFunSuite {
+
+ override def beforeEach(): Unit = {
+ super.beforeEach()
+ CredentialProviderLoader.resetForTesting()
+ }
+
+ private def createSparkConf(): SparkConf = {
+ new SparkConf(loadDefaults = false)
+ .set(SECURITY_OIDC_ENABLED, true)
+ .set(SECURITY_OIDC_IDENTITY_TOKEN_FILE, "/tmp/fake-token")
+ .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 5000L) // 5s for tests
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L) // 1s for tests
+ }
+
+ private def createUserContext(
+ expiresInSeconds: Long = 300): UserContext = {
+ new UserContext(
+ "test-user",
+ "https://issuer.example.com",
+ "fake.jwt.token",
+ Instant.now(),
+ Instant.now().plusSeconds(expiresInSeconds))
+ }
+
+ private def createIngestor(ctx: UserContext): TokenIngestor = {
+ new TokenIngestor {
+ override def load(): Optional[UserContext] = Optional.of(ctx)
+ }
+ }
+
+ private def createFailingIngestor(): TokenIngestor = {
+ new TokenIngestor {
+ override def load(): Optional[UserContext] = Optional.empty()
+ }
+ }
+
+ test("start() acquires initial credentials and invokes callback") {
+ val conf = createSparkConf()
+ val ctx = createUserContext()
+ val callbackRef = new AtomicReference[Array[Byte]]()
+
+ // Use CredentialProviderLoader with the FakeCredentialProvider from
ServiceLoader
+ // FakeCredentialProvider supports scheme "fake"
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ bytes => callbackRef.set(bytes))
+
+ try {
+ val result = manager.start()
+ assert(result != null, "start() should return serialized credentials")
+ assert(callbackRef.get() != null, "callback should have been invoked")
+
+ // Verify deserialization
+ val creds = UserCredentialManager.deserializeUserCredentials(result)
+ assert(creds.forScheme("fake").isPresent,
+ "Should have credentials for 'fake' scheme")
+ assert(creds.forScheme("fake").get().getProperties.get("provider") ===
"fake")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("start() throws when TokenIngestor returns empty (fail-fast)") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ val manager = new UserCredentialManager(
+ conf,
+ createFailingIngestor(),
+ _ => ())
+
+ try {
+ val ex = intercept[IllegalStateException] {
+ manager.start()
+ }
+ assert(ex.getMessage.contains("identity token file is missing or
malformed"))
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("serialization round-trip of UserCredentials") {
+ val props = new util.HashMap[String, String]()
+ props.put("fs.s3a.access.key", "AKIAEXAMPLE")
+ props.put("fs.s3a.secret.key", "secret123")
+ props.put("fs.s3a.session.token", "token456")
+ val cred = new ServiceCredential(props, Instant.now().plusSeconds(3600))
+
+ val credsMap = new util.HashMap[String, ServiceCredential]()
+ credsMap.put("s3a", cred)
+ val original = new UserCredentials(credsMap)
+
+ val conf = createSparkConf()
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+
+ try {
+ val serialized = UserCredentialManager.serializeUserCredentials(original)
+ val deserialized =
UserCredentialManager.deserializeUserCredentials(serialized)
+
+ assert(deserialized.forScheme("s3a").isPresent)
+ val restored = deserialized.forScheme("s3a").get()
+ assert(restored.getProperties.get("fs.s3a.access.key") === "AKIAEXAMPLE")
+ assert(restored.getProperties.get("fs.s3a.secret.key") === "secret123")
+ assert(restored.getProperties.get("fs.s3a.session.token") === "token456")
+ assert(restored.getExpiresAt === cred.getExpiresAt)
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("deserialization rejects unauthorized classes (ObjectInputFilter)") {
+ // Serialize a class that is NOT in the allowed filter pattern.
+ // The filter allows org.apache.spark.security.**, java.util.**,
java.time.**,
+ // java.lang.** but blocks everything else (including java.io.File).
+ val bos = new java.io.ByteArrayOutputStream()
+ val oos = new java.io.ObjectOutputStream(bos)
+ oos.writeObject(new java.io.File("/tmp/malicious"))
+ oos.close()
+
+ val ex = intercept[java.io.InvalidClassException] {
+ UserCredentialManager.deserializeUserCredentials(bos.toByteArray)
+ }
+ // ObjectInputFilter rejects classes not in the allowlist
+ assert(ex.getMessage.contains("REJECTED") ||
ex.getMessage.contains("filter"),
+ s"Expected filter rejection, got: ${ex.getMessage}")
+ }
+
+ test("computeRenewalDelay respects safetyMargin and minInterval") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L) // 5s
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ // Token expires in 60s, credential expires in 30s
+ // Expected: min(60s, 30s) - 10s = 20s
+ val ctx = createUserContext(expiresInSeconds = 60)
+ val credExpiry = Some(Instant.now().plusSeconds(30))
+ val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+ // Allow 1s tolerance for timing
+ assert(delay >= 19000 && delay <= 21000,
+ s"Expected ~20000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay uses minInterval when expiry is very close") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L)
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ // Token expires in 5s, safetyMargin is 10s -> computed delay would be
negative
+ // Should be bounded by minInterval (5s)
+ val ctx = createUserContext(expiresInSeconds = 5)
+ val credExpiry = Some(Instant.now().plusSeconds(5))
+ val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+ assert(delay === 5000L, s"Expected minInterval (5000ms), got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay uses identity token expiry when credential has no
expiry") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L) // 5s
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ // Token expires in 60s, no credential expiry
+ // Expected: 60s - 10s = 50s
+ val ctx = createUserContext(expiresInSeconds = 60)
+ val delay = manager.computeRenewalDelay(ctx, None)
+
+ assert(delay >= 49000 && delay <= 51000,
+ s"Expected ~50000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay returns default when no expiry information") {
+ val conf = createSparkConf()
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ // UserContext with null expiresAt
+ val ctx = new UserContext(
+ "test-user", "https://issuer.example.com", "fake.jwt.token",
+ Instant.now(), null)
+ val delay = manager.computeRenewalDelay(ctx, None)
+
+ // Should be DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS = 7 minutes = 420000ms
+ assert(delay === 420000L, s"Expected 420000ms (7 min), got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay increases exponentially") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ val failuresField =
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // First failure: base = minInterval * 2^0 = 1000ms
+ failuresField.setInt(manager, 1)
+ val delay1 = manager.computeBackoffDelay()
+ assert(delay1 >= 1000 && delay1 <= 1200,
+ s"First backoff should be ~1000-1100ms, got ${delay1}ms")
+
+ // Second failure: base = minInterval * 2^1 = 2000ms
+ failuresField.setInt(manager, 2)
+ val delay2 = manager.computeBackoffDelay()
+ assert(delay2 >= 2000 && delay2 <= 2300,
+ s"Second backoff should be ~2000-2200ms, got ${delay2}ms")
+
+ // Third failure: base = minInterval * 2^2 = 4000ms
+ failuresField.setInt(manager, 3)
+ val delay3 = manager.computeBackoffDelay()
+ assert(delay3 >= 4000 && delay3 <= 4500,
+ s"Third backoff should be ~4000-4400ms, got ${delay3}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay is capped at maxBackoffMs") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ val failuresField =
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // Many failures: should be capped at 10 minutes (600000ms)
+ failuresField.setInt(manager, 20)
+ val delay = manager.computeBackoffDelay()
+ assert(delay <= 660000L, // 600000 + 10% jitter
+ s"Backoff should be capped at ~600000ms + jitter, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay handles zero consecutiveFailures defensively") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _
=> ())
+ try {
+ val failuresField =
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // Edge case: 0 failures (should not happen in practice, but defensive)
+ failuresField.setInt(manager, 0)
+ val delay = manager.computeBackoffDelay()
+ // shiftAmount = max(0, min(0-1, 6)) = max(0, -1) = 0
+ // baseDelay = 1000 * 2^0 = 1000
+ assert(delay >= 1000 && delay <= 1200,
+ s"With 0 failures, backoff should be ~1000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("UserCredentialManager.create returns None when disabled") {
+ val conf = new SparkConf(loadDefaults = false)
+ .set(SECURITY_OIDC_ENABLED, false)
+
+ val result = UserCredentialManager.create(conf, _ => ())
+ assert(result.isEmpty)
+ }
+
+ test("UserCredentialManager.create returns Some when enabled with valid
config") {
+ val conf = createSparkConf()
+ val result = UserCredentialManager.create(conf, _ => ())
+ assert(result.isDefined)
+ }
+
+ test("UserCredentialManager.create throws when enabled without token file") {
+ val conf = new SparkConf(loadDefaults = false)
+ .set(SECURITY_OIDC_ENABLED, true)
+ // Deliberately not setting SECURITY_OIDC_IDENTITY_TOKEN_FILE
+
+ val ex = intercept[IllegalArgumentException] {
+ UserCredentialManager.create(conf, _ => ())
+ }
+ assert(ex.getMessage.contains("spark.security.oidc.identityToken.file"))
+ }
+
+ test("renewal is scheduled after successful credential acquisition") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ val ctx = createUserContext(expiresInSeconds = 60)
+ var callbackCount = 0
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ _ => { callbackCount += 1 })
+
+ try {
+ val result = manager.start()
+ assert(result != null)
+ assert(callbackCount === 1, "callback should be invoked once on start")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("stop() after start() does not throw") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ val ctx = createUserContext()
+
+ val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+ manager.start()
+ // Should not throw
+ manager.stop()
+ }
+
+ test("per-provider error isolation: one provider failure does not block
others") {
+ // Configure two schemes: "fake" (will succeed) and "shared" (will trigger
ambiguity
+ // because no explicit provider is set for it and both
FakeCredentialProvider and
+ // AnotherFakeCredentialProvider support "shared").
+ // The per-provider catch should absorb the IllegalArgumentException for
"shared"
+ // and still return credentials from "fake".
+ val conf = createSparkConf()
+ // Only set explicit provider for "fake", leave "shared" ambiguous
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+ val ctx = createUserContext()
+ val callbackRef = new AtomicReference[Array[Byte]]()
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ bytes => callbackRef.set(bytes))
+
+ try {
+ val result = manager.start()
+ assert(result != null, "start() should return serialized credentials")
+
+ // Verify that "fake" credentials were resolved despite "shared" failing
+ val creds = UserCredentialManager.deserializeUserCredentials(result)
+ assert(creds.forScheme("fake").isPresent,
+ "Should have credentials for 'fake' scheme even though 'shared'
failed")
+ assert(creds.forScheme("fake").get().getProperties.get("provider") ===
"fake")
+ // "shared" should NOT be present (its provider was not found)
+ assert(!creds.forScheme("shared").isPresent,
+ "'shared' scheme should not have credentials due to provider
resolution failure")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("per-provider error isolation: all providers fail throws
IllegalStateException") {
+ // Configure only "shared" with a non-existent provider class
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+ val ctx = createUserContext()
+
+ val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+
+ try {
+ val ex = intercept[IllegalStateException] {
+ manager.start()
+ }
+ assert(ex.getMessage.contains("No credential providers resolved any
credentials"))
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("token rotation triggers new credential acquisition and propagation") {
+ val conf = createSparkConf()
+ .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 1000L) // 1s
+ .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 500L) // 0.5s
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ // First token: user-A
+ val ctx1 = new UserContext(
+ "user-A", "https://issuer.example.com", "token-A",
+ Instant.now(), Instant.now().plusSeconds(2)) // expires in 2s
+
+ // Second token: user-B (simulating rotation)
+ val ctx2 = new UserContext(
+ "user-B", "https://issuer.example.com", "token-B",
+ Instant.now(), Instant.now().plusSeconds(300))
+
+ // TokenIngestor that returns ctx1 initially, then ctx2 after first call
+ val callCount = new AtomicInteger(0)
+ val rotatingIngestor = new TokenIngestor {
+ override def load(): Optional[UserContext] = {
+ if (callCount.getAndIncrement() == 0) Optional.of(ctx1)
+ else Optional.of(ctx2)
+ }
+ }
+
+ val callbacks = new
java.util.concurrent.CopyOnWriteArrayList[Array[Byte]]()
+ val manager = new UserCredentialManager(
+ conf,
+ rotatingIngestor,
+ bytes => callbacks.add(bytes))
+
+ try {
+ val initial = manager.start()
+ assert(initial != null)
+ assert(callbacks.size() === 1, "Should have one callback from start()")
+
+ // Wait for renewal to fire (token expires in 2s, safety margin 1s ->
renews after ~1s)
+ eventually(timeout(10.seconds)) {
+ // After renewal, a second callback should have been invoked with new
credentials
+ assert(callbacks.size() >= 2,
+ s"Expected at least 2 callbacks (initial + renewal), got
${callbacks.size()}")
+ }
+
+ // Verify that the ingestor was called more than once (rotation detected)
+ assert(callCount.get() >= 2,
+ s"TokenIngestor should have been called at least twice, got
${callCount.get()}")
+ } finally {
+ manager.stop()
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]