This is an automated email from the ASF dual-hosted git repository.

sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 86de29d22894 [SPARK-57893][CORE] Implement `UserCredentialManager`
86de29d22894 is described below

commit 86de29d22894b4ad131c6fbdedf953945e2dae24
Author: Kousuke Saruta <[email protected]>
AuthorDate: Wed Jul 29 07:37:47 2026 +0900

    [SPARK-57893][CORE] Implement `UserCredentialManager`
    
    ### What changes were proposed in this pull request?
    This is a part of the SPIP: OIDC Credential Propagation (SPARK-57703).
    This PR adds `UserCredentialManager`, the driver-side credential renewal 
loop.
    
    `UserCredentialManager` is a sibling of `HadoopDelegationTokenManager` and 
handles the OIDC credential propagation path independently. Both managers run 
on independent threads and can both be active simultaneously.
    
    **Responsibilities:**
    1. Reads the current identity token via `TokenIngestor.load()`
    2. Calls `CredentialProvider.resolve()` for each configured scheme
    3. Serializes the resulting `UserCredentials` and invokes the propagation 
callback
    4. Schedules renewal based on `min(identity token expiry, service 
credential expiry) - safetyMargin`
    5. Retries with exponential backoff on failure
    
    **Key design decisions:**
    - **Fail-fast on startup:** If the initial credential acquisition fails, 
the application fails to start rather than running with null credentials.
    - **Per-provider error isolation:** One provider failure does not abort the 
entire resolution loop. Other providers continue to resolve.
    - **Separation of concerns:** Credential *resolution* failures trigger 
backoff; *propagation* failures are logged and retried on the next scheduled 
renewal.
    - **ObjectInputFilter:** Deserialization is restricted to only the classes 
needed for `UserCredentials`, preventing deserialization attacks.
    
    **New files:**
    - `core/.../deploy/security/UserCredentialManager.scala`: Main 
implementation
    - `core/.../deploy/security/UserCredentialManagerSuite.scala`: Unit tests
    
    **Modified files:**
    - `CredentialProviderLoader.java`: Added `discoverAllSchemes()` for 
auto-discovery of registered schemes
    - `config/package.scala`: Added `spark.security.oidc.*` configuration keys
    
    ### Why are the changes needed?
    There is no component to orchestrate credential acquisition, renewal 
scheduling, and propagation triggering on the driver for OIDC-based 
environments. This is the core driver-side engine that coordinates 
`TokenIngestor` and `CredentialProvider` to maintain fresh credentials.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. All new behavior is gated by `spark.security.oidc.enabled=false` 
(default). No existing APIs are changed.
    
    ### How was this patch tested?
    
    Unit tests in `UserCredentialManagerSuite` covering:
    - Initial credential acquisition and callback invocation
    - Fail-fast when TokenIngestor returns empty
    - Serialization/deserialization round-trip with ObjectInputFilter security
    - `computeRenewalDelay` edge cases (safety margin, min interval, no expiry)
    - Exponential backoff (growth, cap, zero-failure edge case)
    - Per-provider error isolation (partial failure, total failure)
    - Token rotation triggers new credential acquisition and propagation
    - Factory method (`create`) with various configurations
    - Lifecycle safety (double-start guard, stop after start)
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Kiro CLI / Claude
    
    Closes #57387 from 
sarutak/oidc-propagation/subtask4-user-credential-manager.
    
    Authored-by: Kousuke Saruta <[email protected]>
    Signed-off-by: Kousuke Saruta <[email protected]>
---
 .../spark/security/CredentialProviderLoader.java   |  33 +-
 .../deploy/security/UserCredentialManager.scala    | 448 +++++++++++++++++++
 .../org/apache/spark/internal/config/package.scala |  40 ++
 .../security/UserCredentialManagerSuite.scala      | 478 +++++++++++++++++++++
 4 files changed, 998 insertions(+), 1 deletion(-)

diff --git 
a/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java 
b/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
index b22a6aed2b55..9d6e9e6fedf0 100644
--- a/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
+++ b/core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java
@@ -20,6 +20,7 @@ package org.apache.spark.security;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.IdentityHashMap;
 import java.util.List;
 import java.util.Locale;
@@ -30,6 +31,8 @@ import java.util.ServiceLoader;
 import java.util.Set;
 import java.util.stream.Collectors;
 
+import com.google.common.annotations.VisibleForTesting;
+
 import org.apache.spark.annotation.Private;
 
 /**
@@ -221,10 +224,38 @@ public final class CredentialProviderLoader {
     return result;
   }
 
+  /**
+   * Discovers all URI schemes supported by providers on the classpath.
+   * <p>
+   * This method queries all discovered {@link CredentialProvider} instances 
and collects
+   * their {@link CredentialProvider#supportedSchemes()} into a single set. 
Unlike
+   * {@link #providerFor(String, Map)}, this does not initialize providers or 
apply
+   * explicit selection rules -- it only reports what schemes are potentially 
available.
+   * <p>
+   * Intended for use by {@code UserCredentialManager} when no explicit scheme 
configuration
+   * (e.g., {@code spark.security.oidc.provider.<scheme>}) is provided.
+   *
+   * @return a set of all supported scheme names (lowercased), possibly empty
+   */
+  public static Set<String> discoverAllSchemes() {
+    List<CredentialProvider> providers = getProviders();
+    Set<String> schemes = new HashSet<>();
+    for (CredentialProvider provider : providers) {
+      Set<String> providerSchemes = provider.supportedSchemes();
+      if (providerSchemes != null) {
+        for (String s : providerSchemes) {
+          schemes.add(s.toLowerCase(Locale.ROOT));
+        }
+      }
+    }
+    return schemes;
+  }
+
   /**
    * Resets the cached provider list and initialization tracking. Intended for 
testing only.
    */
-  static void resetForTesting() {
+  @VisibleForTesting
+  public static void resetForTesting() {
     synchronized (CredentialProviderLoader.class) {
       cachedProviders = null;
       initializedProviders.clear();
diff --git 
a/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
 
b/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
new file mode 100644
index 000000000000..c98771ee0265
--- /dev/null
+++ 
b/core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, 
ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, 
ScheduledExecutorService, TimeUnit}
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the 
OIDC credential
+ * propagation path independently. Both managers run on independent threads 
and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 
via OIDC).
+ *
+ * Responsibilities:
+ *   1. Reads the current identity token via a [[TokenIngestor]]
+ *   2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ *   3. Serializes the resulting [[UserCredentials]] and invokes the 
propagation callback
+ *   4. Schedules renewal based on `min(identity token expiry, service 
credential expiry) -
+ *      safetyMargin`
+ *   5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.oidc.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+    sparkConf: SparkConf,
+    tokenIngestor: TokenIngestor,
+    onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+  private val safetyMargin = sparkConf.get(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN)
+  private val minInterval = sparkConf.get(SECURITY_OIDC_RENEWAL_MIN_INTERVAL)
+
+  // Counter for exponential backoff calculation.
+  // Only accessed from the single-thread renewal executor.
+  private var consecutiveFailures: Int = 0
+  private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+  private var renewalExecutor: ScheduledExecutorService = _
+
+  // Snapshot of credential-related configuration at construction time.
+  private val credentialConfMap: java.util.Map[String, String] = 
sparkConf.getAll
+    .filter(_._1.startsWith("spark.security.oidc."))
+    .toMap.asJava
+
+  /**
+   * Start the credential manager. Acquires initial credentials and schedules 
renewal.
+   *
+   * The initial credential acquisition is fail-fast: if the token cannot be 
loaded or
+   * no credentials can be resolved, an exception is thrown. Subsequent 
renewal failures
+   * are handled with exponential backoff.
+   *
+   * @return The serialized initial [[UserCredentials]].
+   * @throws IllegalStateException if the initial credential acquisition fails.
+   */
+  def start(): Array[Byte] = {
+    require(renewalExecutor == null, "start() must not be called more than 
once")
+
+    // Initial acquisition is fail-fast (no retry/backoff).
+    // This ensures the application fails to start if credentials cannot be 
obtained,
+    // rather than running with null credentials.
+    val userContext = tokenIngestor.load()
+    if (!userContext.isPresent) {
+      throw new IllegalStateException(
+        "Failed to start UserCredentialManager: identity token file is missing 
or malformed. " +
+          s"Check ${SECURITY_OIDC_IDENTITY_TOKEN_FILE.key} configuration.")
+    }
+
+    val ctx = userContext.get()
+    logInfo(log"Initial identity token loaded for principal " +
+      log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+      log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+    val (credentials, earliestExpiry) = resolveCredentials(ctx)
+    val serialized = 
UserCredentialManager.serializeUserCredentials(credentials)
+
+    // Propagate initial credentials
+    onCredentialsUpdate(serialized)
+
+    // Create the renewal executor only after successful initial acquisition.
+    // This avoids leaking a daemon thread if the fail-fast path throws, and
+    // keeps the require() guard valid for retry scenarios.
+    renewalExecutor =
+      
ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+    // Schedule first renewal
+    val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+    scheduleRenewal(renewalDelay)
+
+    logInfo(log"Credential acquisition successful. Next renewal in " +
+      log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+    serialized
+  }
+
+  def stop(): Unit = {
+    if (renewalExecutor != null) {
+      renewalExecutor.shutdownNow()
+    }
+  }
+
+  /**
+   * Scheduled renewal task: load identity token, resolve credentials, 
propagate,
+   * and schedule the next renewal. Failures are retried with exponential 
backoff.
+   */
+  private def renewCredentialsTask(): Unit = {
+    try {
+      val userContext = tokenIngestor.load()
+      if (!userContext.isPresent) {
+        throw new IllegalStateException(
+          "TokenIngestor returned empty - identity token file may be missing 
or malformed")
+      }
+
+      val ctx = userContext.get()
+      logInfo(log"Loaded identity token for principal " +
+        log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+        log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+      val (credentials, earliestExpiry) = resolveCredentials(ctx)
+      val serialized = 
UserCredentialManager.serializeUserCredentials(credentials)
+
+      // Propagate credentials to executors. Errors here are logged separately
+      // so that credential-fetch success is not conflated with distribution 
failure.
+      try {
+        onCredentialsUpdate(serialized)
+      } catch {
+        case e: Exception =>
+          logWarning(log"Credentials were resolved successfully but failed to 
propagate " +
+            log"to executors. Executors will receive updated credentials on 
next renewal.", e)
+      }
+
+      // Reset backoff on successful credential resolution. This is 
intentionally done
+      // even if onCredentialsUpdate failed above: the backoff counter tracks 
credential
+      // *resolution* failures (STS/provider issues), not propagation failures.
+      // Propagation failures are transient and will be retried on next 
scheduled renewal.
+      consecutiveFailures = 0
+
+      // Schedule next renewal
+      val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+      scheduleRenewal(renewalDelay)
+
+      logInfo(log"Credential renewal successful. Next renewal in " +
+        log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+    } catch {
+      case _: InterruptedException =>
+        // Shutting down, ignore
+      case NonFatal(e) =>
+        consecutiveFailures += 1
+        val failures = consecutiveFailures
+        val delay = computeBackoffDelay()
+        logWarning(log"Failed to renew user credentials (attempt " +
+          log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+          log"will retry in ${MDC(LogKeys.TIME_UNITS, 
UIUtils.formatDuration(delay))}.", e)
+        scheduleRenewal(delay)
+      case t: Throwable =>
+        // A fatal error would otherwise be swallowed by the executor, 
silently stopping
+        // all future renewals. Log it before propagating.
+        logError(log"Fatal error during credential renewal; no further 
renewals will be " +
+          log"scheduled.", t)
+        throw t
+    }
+  }
+
+  /**
+   * Resolve credentials for all schemes that have registered providers.
+   *
+   * @return Tuple of (UserCredentials, earliest expiry across all service 
credentials)
+   */
+  private def resolveCredentials(
+      ctx: UserContext): (UserCredentials, Option[Instant]) = {
+    val schemes = discoverSchemes()
+
+    val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+    var earliestExpiry: Option[Instant] = None
+
+    for (scheme <- schemes) {
+      try {
+        val providerOpt = CredentialProviderLoader.providerFor(scheme, 
credentialConfMap)
+        if (providerOpt.isPresent) {
+          val provider = providerOpt.get()
+          // Use a synthetic target URI with just the scheme for initial 
resolution.
+          // The full URI is passed in future versions when path-specific 
resolution is needed.
+          // The "synthetic" authority signals this is not a real endpoint but 
a placeholder
+          // for scheme-based provider selection.
+          val target = new URI(scheme, 
UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+            "/", null, null)
+          val credential = provider.resolve(ctx, target)
+          if (credential == null) {
+            logWarning(log"Provider for scheme ${MDC(LogKeys.URI, scheme)} " +
+              log"returned null; skipping.")
+          } else {
+            credentialMap.put(scheme, credential)
+
+            val expiry = credential.getExpiresAt
+            if (expiry != null) {
+              earliestExpiry = earliestExpiry match {
+                case Some(existing) if existing.isBefore(expiry) => 
Some(existing)
+                case _ => Some(expiry)
+              }
+            }
+          }
+        } else {
+          logWarning(log"No credential provider found for scheme " +
+            log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+        }
+      } catch {
+        case e: Exception =>
+          logWarning(log"Failed to resolve credentials for scheme " +
+            log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+      }
+    }
+
+    if (credentialMap.isEmpty) {
+      throw new IllegalStateException(
+        "No credential providers resolved any credentials. " +
+          "Check that providers are on the classpath and configured 
correctly.")
+    }
+
+    (new UserCredentials(credentialMap.asJava), earliestExpiry)
+  }
+
+  /**
+   * Discover all schemes that have at least one registered provider.
+   *
+   * Schemes are determined by explicit configuration keys of the form
+   * `spark.security.oidc.provider.<scheme>`. If no explicit configuration
+   * exists, the method queries `CredentialProviderLoader` to discover all 
providers
+   * registered via ServiceLoader and collects their supported schemes.
+   *
+   * Note: When using auto-discovery (no explicit config), multiple providers 
may
+   * support the same scheme. In that case, 
`CredentialProviderLoader.providerFor`
+   * will throw an `IllegalArgumentException` for that scheme. The caller
+   * (`resolveCredentials`) handles this gracefully via per-provider exception 
catching,
+   * logging a warning and continuing with remaining schemes.
+   */
+  private def discoverSchemes(): Set[String] = {
+    // Extract explicitly configured scheme names.
+    // Keys are of the form spark.security.oidc.provider.<scheme> or
+    // spark.security.oidc.provider.<scheme>.<subkey>. We extract only the 
first
+    // segment after the prefix to get the scheme name.
+    val providerPrefix = "spark.security.oidc.provider."
+    val explicitSchemes = credentialConfMap.asScala
+      .filter { case (k, _) => k.startsWith(providerPrefix) }
+      .map { case (k, _) => k.stripPrefix(providerPrefix).split('.').head }
+      .toSet
+
+    if (explicitSchemes.nonEmpty) {
+      explicitSchemes
+    } else {
+      // No explicit scheme configuration. Discover all schemes that have a 
provider
+      // available on the classpath by probing CredentialProviderLoader.
+      // This covers both built-in providers (e.g., connector/credential-aws 
for "s3a")
+      // and third-party providers registered via ServiceLoader.
+      CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+    }
+  }
+
+  /**
+   * Compute the delay until next renewal.
+   * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+   * bounded below by minInterval.
+   */
+  private[security] def computeRenewalDelay(
+      ctx: UserContext,
+      earliestCredentialExpiry: Option[Instant]): Long = {
+    val now = System.currentTimeMillis()
+
+    // Consider identity token expiry
+    val tokenExpiry: Option[Long] = 
Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+    // Consider earliest service credential expiry
+    val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+    // Take the minimum of both
+    val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+      case (Some(t), Some(c)) => Some(math.min(t, c))
+      case (Some(t), None) => Some(t)
+      case (None, Some(c)) => Some(c)
+      case (None, None) => None
+    }
+
+    effectiveExpiry match {
+      case Some(expiry) =>
+        math.max(expiry - now - safetyMargin, minInterval)
+      case None =>
+        // No expiry information available from either the identity token or 
service
+        // credentials. Use half of the default suggested TTL (15 min / 2 = 
7.5 min,
+        // rounded down) as a conservative polling interval. This ensures 
credentials
+        // are refreshed even when providers don't report expiration times.
+        UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+    }
+  }
+
+  /**
+   * Compute backoff delay using exponential backoff with jitter.
+   * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+   */
+  private[security] def computeBackoffDelay(): Long = {
+    // Guard against negative or zero shift amounts. consecutiveFailures 
should always
+    // be >= 1 when this is called (incremented before calling), but we 
protect against
+    // edge cases defensively.
+    val shiftAmount = math.max(0, math.min(consecutiveFailures - 1, 6))
+    val baseDelay = minInterval * (1L << shiftAmount)
+    val cappedDelay = math.min(baseDelay, maxBackoffMs)
+    // Add 10% jitter to avoid thundering herd on recovery
+    val jitter = (cappedDelay * 0.1 * math.random()).toLong
+    math.max(cappedDelay + jitter, minInterval)
+  }
+
+  private def scheduleRenewal(delay: Long): Unit = {
+    try {
+      val renewalTask = new Runnable {
+        override def run(): Unit = {
+          renewCredentialsTask()
+        }
+      }
+      renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+    } catch {
+      case _: RejectedExecutionException =>
+        // Executor has been shut down (e.g., stop() called concurrently). 
This is expected
+        // during application shutdown -- no further renewals will be 
scheduled.
+        logDebug(log"Renewal scheduling rejected - executor is shut down.")
+    }
+  }
+
+}
+
+private[spark] object UserCredentialManager {
+
+  /**
+   * Synthetic authority used in target URIs for scheme-based provider 
resolution.
+   * Providers should not rely on this value; it signals that no specific 
endpoint
+   * is targeted and only the URI scheme is meaningful.
+   */
+  private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+  /**
+   * Maximum backoff delay between credential renewal retry attempts.
+   * Caps the exponential backoff to prevent excessively long gaps between 
retries.
+   */
+  private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+  /**
+   * Default renewal interval when neither the identity token nor service 
credentials
+   * report an expiration time. Set to 7 minutes (half of the default 15-minute
+   * suggested TTL from CredentialProvider.suggestedTtl()).
+   */
+  private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = 
TimeUnit.MINUTES.toMillis(7)
+
+  /**
+   * ObjectInputFilter pattern restricting deserialization to only the classes 
needed
+   * for UserCredentials. This prevents deserialization gadget chain attacks 
while
+   * allowing Java's built-in collection internal classes and arrays that 
HashMap uses.
+   */
+  private val DESERIALIZATION_FILTER: String =
+    "org.apache.spark.security.**;" +
+    "java.util.**;" +
+    "java.time.**;" +
+    "java.lang.**;" +
+    "maxdepth=10;" +
+    "maxarray=1000;" +
+    "maxrefs=1000;" +
+    "!*"
+
+  /**
+   * Create a UserCredentialManager if OIDC credential propagation is enabled.
+   *
+   * @param sparkConf The Spark configuration
+   * @param onCredentialsUpdate Callback to propagate credentials to executors
+   * @return Some(manager) if enabled, None otherwise
+   */
+  def create(
+      sparkConf: SparkConf,
+      onCredentialsUpdate: Array[Byte] => Unit): Option[UserCredentialManager] 
= {
+    if (!sparkConf.get(SECURITY_OIDC_ENABLED)) {
+      None
+    } else {
+      val tokenFile = 
sparkConf.get(SECURITY_OIDC_IDENTITY_TOKEN_FILE).getOrElse {
+        throw new IllegalArgumentException(
+          s"${SECURITY_OIDC_IDENTITY_TOKEN_FILE.key} must be set when " +
+            s"${SECURITY_OIDC_ENABLED.key} is true")
+      }
+
+      val tokenIngestor = new FileTokenIngestor(Paths.get(tokenFile))
+      Some(new UserCredentialManager(sparkConf, tokenIngestor, 
onCredentialsUpdate))
+    }
+  }
+
+  /**
+   * Serialize [[UserCredentials]] to a byte array using Java serialization.
+   */
+  private[security] def serializeUserCredentials(credentials: 
UserCredentials): Array[Byte] = {
+    val bos = new ByteArrayOutputStream()
+    Utils.tryWithResource(new ObjectOutputStream(bos)) { oos =>
+      oos.writeObject(credentials)
+      oos.flush()
+    }
+    bos.toByteArray
+  }
+
+  /**
+   * Deserialize [[UserCredentials]] from a byte array.
+   *
+   * Uses an [[ObjectInputFilter]] to restrict deserialized classes to only 
those
+   * required for [[UserCredentials]], preventing deserialization attacks.
+   */
+  def deserializeUserCredentials(bytes: Array[Byte]): UserCredentials = {
+    val bis = new java.io.ByteArrayInputStream(bytes)
+    Utils.tryWithResource(new ObjectInputStream(bis)) { ois =>
+      ois.setObjectInputFilter(
+        ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
+      ois.readObject().asInstanceOf[UserCredentials]
+    }
+  }
+}
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala 
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index a5f74d5b7bed..02e0fc9f0089 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1690,6 +1690,46 @@ package object config {
       .timeConf(TimeUnit.SECONDS)
       .createWithDefaultString("1h")
 
+  private[spark] val SECURITY_OIDC_ENABLED =
+    ConfigBuilder("spark.security.oidc.enabled")
+      .doc("Whether to enable OIDC credential propagation. When enabled, the 
driver reads an " +
+        "identity token from a file, exchanges it for short-lived service 
credentials via " +
+        "CredentialProvider implementations, and propagates those credentials 
to executors.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(false)
+
+  private[spark] val SECURITY_OIDC_IDENTITY_TOKEN_FILE =
+    ConfigBuilder("spark.security.oidc.identityToken.file")
+      .doc("Path to the OIDC identity token file on the driver. Required when 
" +
+        "spark.security.oidc.enabled is true. The file should contain a JWT 
token " +
+        "(e.g., a Kubernetes projected service account token).")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .stringConf
+      .createOptional
+
+  private[spark] val SECURITY_OIDC_RENEWAL_SAFETY_MARGIN =
+    ConfigBuilder("spark.security.oidc.renewal.safetyMargin")
+      .doc("How long before credential expiry to trigger renewal. Credentials 
are refreshed " +
+        "at min(identity token expiry, service credential expiry) minus this 
margin.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .timeConf(TimeUnit.MILLISECONDS)
+      .checkValue(_ > 0, "The safety margin must be a positive time value.")
+      .createWithDefaultString("60s")
+
+  private[spark] val SECURITY_OIDC_RENEWAL_MIN_INTERVAL =
+    ConfigBuilder("spark.security.oidc.renewal.minInterval")
+      .doc("Minimum interval between credential renewal attempts. This 
prevents tight renewal " +
+        "loops when credentials have very short TTLs or when failures cause 
rapid retries.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .timeConf(TimeUnit.MILLISECONDS)
+      .checkValue(_ > 0, "The minimum renewal interval must be a positive time 
value.")
+      .createWithDefaultString("30s")
+
   private[spark] val SHUFFLE_SORT_INIT_BUFFER_SIZE =
     ConfigBuilder("spark.shuffle.sort.initialBufferSize")
       .internal()
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
 
b/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
new file mode 100644
index 000000000000..b5ee471013b3
--- /dev/null
+++ 
b/core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala
@@ -0,0 +1,478 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.time.Instant
+import java.util
+import java.util.Optional
+import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
+
+import scala.concurrent.duration._
+
+import org.scalatest.concurrent.Eventually.{eventually, timeout}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+
+class UserCredentialManagerSuite extends SparkFunSuite {
+
+  override def beforeEach(): Unit = {
+    super.beforeEach()
+    CredentialProviderLoader.resetForTesting()
+  }
+
+  private def createSparkConf(): SparkConf = {
+    new SparkConf(loadDefaults = false)
+      .set(SECURITY_OIDC_ENABLED, true)
+      .set(SECURITY_OIDC_IDENTITY_TOKEN_FILE, "/tmp/fake-token")
+      .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 5000L) // 5s for tests
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)  // 1s for tests
+  }
+
+  private def createUserContext(
+      expiresInSeconds: Long = 300): UserContext = {
+    new UserContext(
+      "test-user",
+      "https://issuer.example.com";,
+      "fake.jwt.token",
+      Instant.now(),
+      Instant.now().plusSeconds(expiresInSeconds))
+  }
+
+  private def createIngestor(ctx: UserContext): TokenIngestor = {
+    new TokenIngestor {
+      override def load(): Optional[UserContext] = Optional.of(ctx)
+    }
+  }
+
+  private def createFailingIngestor(): TokenIngestor = {
+    new TokenIngestor {
+      override def load(): Optional[UserContext] = Optional.empty()
+    }
+  }
+
+  test("start() acquires initial credentials and invokes callback") {
+    val conf = createSparkConf()
+    val ctx = createUserContext()
+    val callbackRef = new AtomicReference[Array[Byte]]()
+
+    // Use CredentialProviderLoader with the FakeCredentialProvider from 
ServiceLoader
+    // FakeCredentialProvider supports scheme "fake"
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+
+    val manager = new UserCredentialManager(
+      conf,
+      createIngestor(ctx),
+      bytes => callbackRef.set(bytes))
+
+    try {
+      val result = manager.start()
+      assert(result != null, "start() should return serialized credentials")
+      assert(callbackRef.get() != null, "callback should have been invoked")
+
+      // Verify deserialization
+      val creds = UserCredentialManager.deserializeUserCredentials(result)
+      assert(creds.forScheme("fake").isPresent,
+        "Should have credentials for 'fake' scheme")
+      assert(creds.forScheme("fake").get().getProperties.get("provider") === 
"fake")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("start() throws when TokenIngestor returns empty (fail-fast)") {
+    val conf = createSparkConf()
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+
+    val manager = new UserCredentialManager(
+      conf,
+      createFailingIngestor(),
+      _ => ())
+
+    try {
+      val ex = intercept[IllegalStateException] {
+        manager.start()
+      }
+      assert(ex.getMessage.contains("identity token file is missing or 
malformed"))
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("serialization round-trip of UserCredentials") {
+    val props = new util.HashMap[String, String]()
+    props.put("fs.s3a.access.key", "AKIAEXAMPLE")
+    props.put("fs.s3a.secret.key", "secret123")
+    props.put("fs.s3a.session.token", "token456")
+    val cred = new ServiceCredential(props, Instant.now().plusSeconds(3600))
+
+    val credsMap = new util.HashMap[String, ServiceCredential]()
+    credsMap.put("s3a", cred)
+    val original = new UserCredentials(credsMap)
+
+    val conf = createSparkConf()
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+
+    try {
+      val serialized = UserCredentialManager.serializeUserCredentials(original)
+      val deserialized = 
UserCredentialManager.deserializeUserCredentials(serialized)
+
+      assert(deserialized.forScheme("s3a").isPresent)
+      val restored = deserialized.forScheme("s3a").get()
+      assert(restored.getProperties.get("fs.s3a.access.key") === "AKIAEXAMPLE")
+      assert(restored.getProperties.get("fs.s3a.secret.key") === "secret123")
+      assert(restored.getProperties.get("fs.s3a.session.token") === "token456")
+      assert(restored.getExpiresAt === cred.getExpiresAt)
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("deserialization rejects unauthorized classes (ObjectInputFilter)") {
+    // Serialize a class that is NOT in the allowed filter pattern.
+    // The filter allows org.apache.spark.security.**, java.util.**, 
java.time.**,
+    // java.lang.** but blocks everything else (including java.io.File).
+    val bos = new java.io.ByteArrayOutputStream()
+    val oos = new java.io.ObjectOutputStream(bos)
+    oos.writeObject(new java.io.File("/tmp/malicious"))
+    oos.close()
+
+    val ex = intercept[java.io.InvalidClassException] {
+      UserCredentialManager.deserializeUserCredentials(bos.toByteArray)
+    }
+    // ObjectInputFilter rejects classes not in the allowlist
+    assert(ex.getMessage.contains("REJECTED") || 
ex.getMessage.contains("filter"),
+      s"Expected filter rejection, got: ${ex.getMessage}")
+  }
+
+  test("computeRenewalDelay respects safetyMargin and minInterval") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L)   // 5s
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      // Token expires in 60s, credential expires in 30s
+      // Expected: min(60s, 30s) - 10s = 20s
+      val ctx = createUserContext(expiresInSeconds = 60)
+      val credExpiry = Some(Instant.now().plusSeconds(30))
+      val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+      // Allow 1s tolerance for timing
+      assert(delay >= 19000 && delay <= 21000,
+        s"Expected ~20000ms, got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeRenewalDelay uses minInterval when expiry is very close") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L)
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L)
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      // Token expires in 5s, safetyMargin is 10s -> computed delay would be 
negative
+      // Should be bounded by minInterval (5s)
+      val ctx = createUserContext(expiresInSeconds = 5)
+      val credExpiry = Some(Instant.now().plusSeconds(5))
+      val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+      assert(delay === 5000L, s"Expected minInterval (5000ms), got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeRenewalDelay uses identity token expiry when credential has no 
expiry") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 5000L)   // 5s
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      // Token expires in 60s, no credential expiry
+      // Expected: 60s - 10s = 50s
+      val ctx = createUserContext(expiresInSeconds = 60)
+      val delay = manager.computeRenewalDelay(ctx, None)
+
+      assert(delay >= 49000 && delay <= 51000,
+        s"Expected ~50000ms, got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeRenewalDelay returns default when no expiry information") {
+    val conf = createSparkConf()
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      // UserContext with null expiresAt
+      val ctx = new UserContext(
+        "test-user", "https://issuer.example.com";, "fake.jwt.token",
+        Instant.now(), null)
+      val delay = manager.computeRenewalDelay(ctx, None)
+
+      // Should be DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS = 7 minutes = 420000ms
+      assert(delay === 420000L, s"Expected 420000ms (7 min), got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeBackoffDelay increases exponentially") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      val failuresField = 
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+      failuresField.setAccessible(true)
+
+      // First failure: base = minInterval * 2^0 = 1000ms
+      failuresField.setInt(manager, 1)
+      val delay1 = manager.computeBackoffDelay()
+      assert(delay1 >= 1000 && delay1 <= 1200,
+        s"First backoff should be ~1000-1100ms, got ${delay1}ms")
+
+      // Second failure: base = minInterval * 2^1 = 2000ms
+      failuresField.setInt(manager, 2)
+      val delay2 = manager.computeBackoffDelay()
+      assert(delay2 >= 2000 && delay2 <= 2300,
+        s"Second backoff should be ~2000-2200ms, got ${delay2}ms")
+
+      // Third failure: base = minInterval * 2^2 = 4000ms
+      failuresField.setInt(manager, 3)
+      val delay3 = manager.computeBackoffDelay()
+      assert(delay3 >= 4000 && delay3 <= 4500,
+        s"Third backoff should be ~4000-4400ms, got ${delay3}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeBackoffDelay is capped at maxBackoffMs") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      val failuresField = 
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+      failuresField.setAccessible(true)
+
+      // Many failures: should be capped at 10 minutes (600000ms)
+      failuresField.setInt(manager, 20)
+      val delay = manager.computeBackoffDelay()
+      assert(delay <= 660000L, // 600000 + 10% jitter
+        s"Backoff should be capped at ~600000ms + jitter, got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("computeBackoffDelay handles zero consecutiveFailures defensively") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 1000L)
+
+    val manager = new UserCredentialManager(conf, createFailingIngestor(), _ 
=> ())
+    try {
+      val failuresField = 
classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+      failuresField.setAccessible(true)
+
+      // Edge case: 0 failures (should not happen in practice, but defensive)
+      failuresField.setInt(manager, 0)
+      val delay = manager.computeBackoffDelay()
+      // shiftAmount = max(0, min(0-1, 6)) = max(0, -1) = 0
+      // baseDelay = 1000 * 2^0 = 1000
+      assert(delay >= 1000 && delay <= 1200,
+        s"With 0 failures, backoff should be ~1000ms, got ${delay}ms")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("UserCredentialManager.create returns None when disabled") {
+    val conf = new SparkConf(loadDefaults = false)
+      .set(SECURITY_OIDC_ENABLED, false)
+
+    val result = UserCredentialManager.create(conf, _ => ())
+    assert(result.isEmpty)
+  }
+
+  test("UserCredentialManager.create returns Some when enabled with valid 
config") {
+    val conf = createSparkConf()
+    val result = UserCredentialManager.create(conf, _ => ())
+    assert(result.isDefined)
+  }
+
+  test("UserCredentialManager.create throws when enabled without token file") {
+    val conf = new SparkConf(loadDefaults = false)
+      .set(SECURITY_OIDC_ENABLED, true)
+    // Deliberately not setting SECURITY_OIDC_IDENTITY_TOKEN_FILE
+
+    val ex = intercept[IllegalArgumentException] {
+      UserCredentialManager.create(conf, _ => ())
+    }
+    assert(ex.getMessage.contains("spark.security.oidc.identityToken.file"))
+  }
+
+  test("renewal is scheduled after successful credential acquisition") {
+    val conf = createSparkConf()
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+    val ctx = createUserContext(expiresInSeconds = 60)
+    var callbackCount = 0
+
+    val manager = new UserCredentialManager(
+      conf,
+      createIngestor(ctx),
+      _ => { callbackCount += 1 })
+
+    try {
+      val result = manager.start()
+      assert(result != null)
+      assert(callbackCount === 1, "callback should be invoked once on start")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("stop() after start() does not throw") {
+    val conf = createSparkConf()
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+    val ctx = createUserContext()
+
+    val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+    manager.start()
+    // Should not throw
+    manager.stop()
+  }
+
+  test("per-provider error isolation: one provider failure does not block 
others") {
+    // Configure two schemes: "fake" (will succeed) and "shared" (will trigger 
ambiguity
+    // because no explicit provider is set for it and both 
FakeCredentialProvider and
+    // AnotherFakeCredentialProvider support "shared").
+    // The per-provider catch should absorb the IllegalArgumentException for 
"shared"
+    // and still return credentials from "fake".
+    val conf = createSparkConf()
+    // Only set explicit provider for "fake", leave "shared" ambiguous
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+    conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+    val ctx = createUserContext()
+    val callbackRef = new AtomicReference[Array[Byte]]()
+
+    val manager = new UserCredentialManager(
+      conf,
+      createIngestor(ctx),
+      bytes => callbackRef.set(bytes))
+
+    try {
+      val result = manager.start()
+      assert(result != null, "start() should return serialized credentials")
+
+      // Verify that "fake" credentials were resolved despite "shared" failing
+      val creds = UserCredentialManager.deserializeUserCredentials(result)
+      assert(creds.forScheme("fake").isPresent,
+        "Should have credentials for 'fake' scheme even though 'shared' 
failed")
+      assert(creds.forScheme("fake").get().getProperties.get("provider") === 
"fake")
+      // "shared" should NOT be present (its provider was not found)
+      assert(!creds.forScheme("shared").isPresent,
+        "'shared' scheme should not have credentials due to provider 
resolution failure")
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("per-provider error isolation: all providers fail throws 
IllegalStateException") {
+    // Configure only "shared" with a non-existent provider class
+    val conf = createSparkConf()
+    conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+    val ctx = createUserContext()
+
+    val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+
+    try {
+      val ex = intercept[IllegalStateException] {
+        manager.start()
+      }
+      assert(ex.getMessage.contains("No credential providers resolved any 
credentials"))
+    } finally {
+      manager.stop()
+    }
+  }
+
+  test("token rotation triggers new credential acquisition and propagation") {
+    val conf = createSparkConf()
+      .set(SECURITY_OIDC_RENEWAL_SAFETY_MARGIN, 1000L) // 1s
+      .set(SECURITY_OIDC_RENEWAL_MIN_INTERVAL, 500L)   // 0.5s
+    conf.set("spark.security.oidc.provider.fake",
+      "org.apache.spark.security.FakeCredentialProvider")
+
+    // First token: user-A
+    val ctx1 = new UserContext(
+      "user-A", "https://issuer.example.com";, "token-A",
+      Instant.now(), Instant.now().plusSeconds(2)) // expires in 2s
+
+    // Second token: user-B (simulating rotation)
+    val ctx2 = new UserContext(
+      "user-B", "https://issuer.example.com";, "token-B",
+      Instant.now(), Instant.now().plusSeconds(300))
+
+    // TokenIngestor that returns ctx1 initially, then ctx2 after first call
+    val callCount = new AtomicInteger(0)
+    val rotatingIngestor = new TokenIngestor {
+      override def load(): Optional[UserContext] = {
+        if (callCount.getAndIncrement() == 0) Optional.of(ctx1)
+        else Optional.of(ctx2)
+      }
+    }
+
+    val callbacks = new 
java.util.concurrent.CopyOnWriteArrayList[Array[Byte]]()
+    val manager = new UserCredentialManager(
+      conf,
+      rotatingIngestor,
+      bytes => callbacks.add(bytes))
+
+    try {
+      val initial = manager.start()
+      assert(initial != null)
+      assert(callbacks.size() === 1, "Should have one callback from start()")
+
+      // Wait for renewal to fire (token expires in 2s, safety margin 1s -> 
renews after ~1s)
+      eventually(timeout(10.seconds)) {
+        // After renewal, a second callback should have been invoked with new 
credentials
+        assert(callbacks.size() >= 2,
+          s"Expected at least 2 callbacks (initial + renewal), got 
${callbacks.size()}")
+      }
+
+      // Verify that the ingestor was called more than once (rotation detected)
+      assert(callCount.get() >= 2,
+        s"TokenIngestor should have been called at least twice, got 
${callCount.get()}")
+    } finally {
+      manager.stop()
+    }
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to