sarutak opened a new pull request, #58574: URL: https://github.com/apache/spark/pull/58574
### What changes were proposed in this pull request? Split OIDC credential propagation into a provider selection phase and a credential resolution phase, so that provider-declared properties (`CredentialProvider.additionalSparkProperties()`) take effect on the driver, not only on executors. - New selection phase `UserCredentialManager.applyProviderProperties(sparkConf)`: I/O-free (provider discovery + `init()` + `additionalSparkProperties()`, no `resolve()`). `SparkContext` calls it just before it materializes the driver's Hadoop `Configuration`, so the applied `spark.*` properties are picked up by the driver. It is type-agnostic (any `spark.*` property, not only `spark.hadoop.*`) and limited to unambiguously-resolvable schemes, so only providers the job actually uses contribute properties. - Resolution phase unchanged: `UserCredentialManager.start()` still runs on the scheduler backend and performs token load, `resolve()`, executor propagation, and renewal. - The `CredentialProviderLoader` from the selection phase is retained on `SparkContext` and injected into `UserCredentialManager`, so providers are discovered and initialized once. `SparkContext.stop()` closes it (idempotent with the manager's own close) to avoid leaking provider resources in local mode or on a construction-failure path. - Removed the now-redundant resolution-phase auto-config loop and dropped `activeProviders` from `resolveCredentials()`. - Clarified the `CredentialProvider.additionalSparkProperties()` contract: called at selection time, independent of the resolve result, applied once on both driver and executors before consuming components initialize. ### Why are the changes needed? With OIDC enabled and no explicit `fs.s3a.aws.credentials.provider` (the documented usage), the provider-declared S3A wiring reached executors but not the driver, because it was applied after the driver's Hadoop `Configuration` had already been materialized. Driver-side storage access then fell back to the default credential chain (e.g. the node instance profile), causing 403s or access under the wrong identity. This completes a goal the SPIP (SPARK-57703) already stated: "a Spark driver on Kubernetes accesses S3 as the pod's service account, not as the user who submitted the job" is the problem it set out to solve. The initial implementation delivered the executor propagation mechanism, but the wiring that lets the driver itself use the resolved credentials was applied too late in initialization. This PR closes that gap. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `UserCredentialManagerSuite`: added selection-phase tests (type-agnostic application via a non-Hadoop property, no-op when disabled, no override of user-set values, ambiguous-scheme skip, throwing-provider survival). 27 tests pass. - `OidcCredentialIntegrationSuite` (13) and the `credential-aws` module tests (69) pass. - `OidcCredentialE2ESuite` no longer sets `fs.s3a.aws.credentials.provider` explicitly, so all scenarios exercise the driver-side auto-config path; added an "explicit provider is respected" test. - Manually verified end to end on real AWS EKS + STS + S3 (cluster mode, no explicit provider): the driver acquires OIDC credentials, reads/writes S3, and propagates to executors. ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
