sarutak opened a new pull request, #57387: URL: https://github.com/apache/spark/pull/57387
### What changes were proposed in this pull request? This is a part of the SPIP: OIDC Credential Propagation (SPARK-57703). This PR adds `UserCredentialManager`, the driver-side credential renewal loop. `UserCredentialManager` is a sibling of `HadoopDelegationTokenManager` and handles the OIDC credential propagation path independently. Both managers run on independent threads and can both be active simultaneously. **Responsibilities:** 1. Reads the current identity token via `TokenIngestor.load()` 2. Calls `CredentialProvider.resolve()` for each configured scheme 3. Serializes the resulting `UserCredentials` and invokes the propagation callback 4. Schedules renewal based on `min(identity token expiry, service credential expiry) - safetyMargin` 5. Retries with exponential backoff on failure **Key design decisions:** - **Fail-fast on startup:** If the initial credential acquisition fails, the application fails to start rather than running with null credentials. - **Per-provider error isolation:** One provider failure does not abort the entire resolution loop. Other providers continue to resolve. - **Separation of concerns:** Credential *resolution* failures trigger backoff; *propagation* failures are logged and retried on the next scheduled renewal. - **ObjectInputFilter:** Deserialization is restricted to only the classes needed for `UserCredentials`, preventing deserialization attacks. **New files:** - `core/.../deploy/security/UserCredentialManager.scala` — Main implementation - `core/.../deploy/security/UserCredentialManagerSuite.scala` — Unit tests **Modified files:** - `CredentialProviderLoader.java` — Added `discoverAllSchemes()` for auto-discovery of registered schemes - `config/package.scala` — Added `spark.security.credentials.*` configuration keys - `configs-without-binding-policy-exceptions` — Registered new config keys ### Why are the changes needed? There is no component to orchestrate credential acquisition, renewal scheduling, and propagation triggering on the driver for OIDC-based environments. This is the core driver-side engine that coordinates `TokenIngestor` and `CredentialProvider` to maintain fresh credentials. ### Does this PR introduce _any_ user-facing change? No. All new behavior is gated by `spark.security.credentials.enabled=false` (default). No existing APIs are changed. ### How was this patch tested? Unit tests in `UserCredentialManagerSuite` covering: - Initial credential acquisition and callback invocation - Fail-fast when TokenIngestor returns empty - Serialization/deserialization round-trip with ObjectInputFilter security - `computeRenewalDelay` edge cases (safety margin, min interval, no expiry) - Exponential backoff (growth, cap, zero-failure edge case) - Per-provider error isolation (partial failure, total failure) - Token rotation triggers new credential acquisition and propagation - Factory method (`create`) with various configurations - Lifecycle safety (double-start guard, stop after start) ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
