yadavay-amzn opened a new pull request, #57191: URL: https://github.com/apache/spark/pull/57191
### What changes were proposed in this pull request? This is task 2 of the OIDC Credential Propagation SPIP ([SPARK-57703](https://issues.apache.org/jira/browse/SPARK-57703)), building on the core types added in SPARK-57890. It introduces the pluggable provider SPI in `org.apache.spark.security` (spark-core): - `CredentialProvider` - a `@DeveloperApi` interface that providers (AWS STS, Azure, GCP, etc.) implement to exchange a user identity for a short-lived `ServiceCredential`: ```java @DeveloperApi public interface CredentialProvider { void init(Map<String, String> conf); Set<String> supportedSchemes(); ServiceCredential resolve(UserContext user, URI target) throws CredentialResolutionException; default Duration suggestedTtl() { return Duration.ofMinutes(15); } } ``` - `CredentialResolutionException` - a `@DeveloperApi` checked exception thrown by `resolve`. - `CredentialProviderLoader` - discovers implementations via `ServiceLoader` and selects a provider per scheme. Scheme keys are normalized to lowercase. Selection follows an explicit-configuration policy: `spark.security.credentials.provider.<scheme>` names the fully-qualified provider class to use for that scheme. If exactly one discovered provider supports a scheme, no configuration is needed; if multiple support it and the conf is unset, a clear error is raised listing the candidates. Each selected provider is initialized exactly once. A `FakeCredentialProvider` (and a second one sharing a scheme) plus a `META-INF/services` registration are added under `core/src/test` to exercise discovery and selection. ### Why are the changes needed? There is currently no pluggable SPI for exchanging an identity token for short-lived service credentials. This is the extension point the credential-propagation framework builds on. See the SPIP design document, Appendix A. ### Does this PR introduce _any_ user-facing change? No behavior change. It adds new `@DeveloperApi` types only; nothing wires them up until the follow-up (SPARK-57892 / the manager task). ### How was this patch tested? New JUnit `CredentialProviderLoaderSuite` covering the acceptance criteria and edge cases: `ServiceLoader` discovery, single-candidate selection, explicit-conf selection, multiple-without-conf ambiguity error, bad/unknown configured class, unknown scheme, empty-string conf treated as unset, case-insensitive scheme matching, `init` invoked exactly once (same instance on repeat resolution), `resolve` success, `CredentialResolutionException` propagation, null-input guards, and a clear error when a provider returns `null` from `supportedSchemes()`. ### Design notes (feedback welcome) Two choices worth a look, both easy to change before this lands: - **Binding policy**: explicit selection via `spark.security.credentials.provider.<scheme>` (single fully-qualified class name). This can be extended to an ordered-priority list later without breaking the conf if preferred. - **`init` lifecycle**: each provider is initialized once, on first selection, with that call's configuration (first-conf-wins), inside the loader's lock. This sets the provider lifecycle the manager task will consume. ### Was this patch authored or co-authored using generative AI tooling? Authored with assistance by Claude Opus 4.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
