yadavay-amzn opened a new pull request, #57655: URL: https://github.com/apache/spark/pull/57655
### What changes were proposed in this pull request? Adds a new opt-in connector module `connector/credential-aws` containing `AwsStsCredentialProvider`, which implements the `CredentialProvider` SPI (SPARK-57891) to exchange an OIDC identity token for temporary AWS credentials via STS `AssumeRoleWithWebIdentity`. It returns S3A-compatible credentials (`fs.s3a.access.key` / `fs.s3a.secret.key` / `fs.s3a.session.token`) as a `ServiceCredential`. - Configuration under `spark.security.oidc.*`: `roleArn` (required), `sessionName`, `durationSeconds`, `region`, `stsEndpoint`. - Supports non-AWS STS-compatible endpoints (MinIO, Ceph, LocalStack) via `stsEndpoint`. - Uses an anonymous STS client (web-identity needs no base AWS credentials). - Thread-safe: configuration is resolved once in `init()` and published via a single `volatile` immutable holder read by `resolve()`. - Never exposes the OIDC raw token in logs or exception messages. This is the driver-side provider only (OIDC SPIP task 9). The executor-side `AwsCredentialsProvider` is tracked separately as SPARK-57899 (task 10). ### Why are the changes needed? Part of the OIDC Credential Propagation SPIP ([SPARK-57703](https://issues.apache.org/jira/browse/SPARK-57703)). No `CredentialProvider` existed to exchange an OIDC token for AWS STS temporary credentials; this is the S3/STS reference provider. ### Does this PR introduce any user-facing change? No change to defaults (`spark.security.oidc.enabled` remains off). Adds a new optional module and provider plus the `spark.security.oidc.*` AWS configuration keys. ### How was this patch tested? New `AwsStsCredentialProviderSuite` (JUnit 5 + Mockito, 21 tests) with a mocked `StsClient`: - Successful resolve with `ArgumentCaptor` assertions on the STS request (role ARN, web identity token, session name, duration) and on the mapped `fs.s3a.*` keys + expiration. - STS failure wrapped in `CredentialResolutionException` (asserts the token never appears in the message chain). - Incomplete STS credentials (null credentials / missing session token) throw `CredentialResolutionException` rather than NPE. - Missing/blank `roleArn`, null/blank token, `resolve()` before `init()`. - Resolved region/endpoint for endpoint+region, endpoint-only (default region), and neither. - Session-name derivation, duration handling, `suggestedTtl`, `supportedSchemes`, and `ServiceLoader` discovery. ### Was this patch authored or co-authored using generative AI tooling? Yes. Authored with assistance by Claude Opus 5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
