shrirangmhalgi opened a new pull request, #58018:
URL: https://github.com/apache/spark/pull/58018
### What changes were proposed in this pull request?
Adds a default method additionalSparkProperties() to the CredentialProvider
interface that allows provider modules to declare Spark configuration
properties needed when the provider is active. After successful credential
acquisition, UserCredentialManager.start() iterates all discovered providers
and applies their declared properties to SparkConf -- only if the user has not
already set them explicitly.
AwsStsCredentialProvider overrides this to return
{"spark.hadoop.fs.s3a.aws.credentials.provider":
"org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider"}, enabling
zero-config executor-side S3A credential resolution when OIDC is enabled.
Also adds CredentialProviderLoader.discoverAllProviders() for querying
provider metadata without triggering initialization.
### Why are the changes needed?
This is a Sub-task of the OIDC Credential Propagation SPIP. PR #57998
(SparkOidcAwsCredentialsProvider) originally placed auto-config logic in
CoarseGrainedSchedulerBackend, which was a layer violation -- core referencing
an optional module's class name. This PR moves the mechanism to a generic SPI
where each provider module declares its own properties, keeping vendor-specific
knowledge out of core.
### Does this PR introduce _any_ user-facing change?
Yes. When spark.security.oidc.enabled=true and AwsStsCredentialProvider is
on the classpath, executors automatically use SparkOidcAwsCredentialsProvider
for S3A access without requiring manual Hadoop configuration. User-provided
values are never overwritten.
### How was this patch tested?
AwsStsCredentialProviderSuite (53 tests) passes with the new override
CoarseGrainedSchedulerBackendSuite (24 tests) passes
OidcCredentialIntegrationSuite (13 tests) passes -- exercises the full
credential acquisition flow that triggers additionalSparkProperties()
application
Tested with both Maven and SBT.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
