shrirangmhalgi opened a new pull request, #57998: URL: https://github.com/apache/spark/pull/57998
### What changes were proposed in this pull request? Adds `SparkOidcAwsCredentialsProvider`, an executor-side `AwsCredentialsProvider` (AWS SDK v2) implementation in the `credential-aws` module that reads S3A credentials dynamically from Spark's executor credential store on every `resolveCredentials()` call. The provider: - Reads the latest `VersionedCredentials` from `SparkEnv.userCredentials` (no internal caching) - Deserializes to `UserCredentials`, extracts the `ServiceCredential` for scheme `"s3a"` - Returns `AwsSessionCredentials` with `fs.s3a.access.key`, `fs.s3a.secret.key`, `fs.s3a.session.token` - Throws `IllegalStateException` with actionable error messages for misconfiguration - Wraps deserialization failures with credential version number for operational debugging Additionally, `CoarseGrainedSchedulerBackend.setupUserCredentialManager()` now auto-sets `spark.hadoop.fs.s3a.aws.credentials.provider` when OIDC is enabled and the user has not explicitly configured a provider. This propagates to executors via `SparkAppConfig`. No new dependencies are introduced - `software.amazon.awssdk:auth` is already available transitively from the existing sts dependency. ### Why are the changes needed? This is a Sub-task of the OIDC Credential Propagation SPIP. The driver-side credential acquisition (`AwsStsCredentialProvider`) and executor-side credential delivery (`VersionedCredentials` store) are already merged. This PR adds the bridge between Spark's credential store and S3A's `AwsCredentialsProvider` interface on executors. Without this, users would need to implement their own adapter to read from `SparkEnv.userCredentials` and translate to AWS SDK v2 credentials. This class provides that bridge as a first-party, zero-config solution. ### Does this PR introduce _any_ user-facing change? Yes. When `spark.security.oidc.enabled=true`, executors automatically use `SparkOidcAwsCredentialsProvider` for S3A access - no manual Hadoop configuration required. Users can also configure it explicitly: `fs.s3a.aws.credentials.provider=org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider` User-provided `fs.s3a.aws.credentials.provider` values are never overwritten. ### How was this patch tested? Added SparkOidcAwsCredentialsProviderSuite with 16 tests covering: - Happy path: valid credentials returned as `AwsSessionCredentials` - Dynamic refresh: store updates immediately visible (no caching) - No-stale guarantee: 10 consecutive reads after refresh all return latest - Error cases: null SparkEnv, empty store, missing s3a scheme, missing access key/secret key/session token (all with message assertions) - End-to-end serialization roundtrip (real `ObjectOutputStream` / `ObjectInputStream` path) - Version guard: stale v1 rejected after v2 arrives (uses real `VersionedCredentials.updateIfNewer`) - Scheme normalization, expired credential passthrough, multi-scheme bundles - Interface contract verification - Rapid store update consistency (100 iterations) All 69 tests in the `credential-aws` module pass (16 new + 53 existing `AwsStsCredentialProviderSuite`). `CoarseGrainedSchedulerBackendSuite` (24 tests) passes with the auto-config change. Tested with both Maven and SBT. ### Was this patch authored or co-authored using generative AI tooling? Yes. CoAuthored using Claude Opus 4.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
