[
https://issues.apache.org/jira/browse/SPARK-58789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shrirang Mhalgi updated SPARK-58789:
------------------------------------
Shepherd: Kousuke Saruta
> Add CredentialProvider.additionalSparkProperties() SPI for executor-side
> auto-configuration
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-58789
> URL: https://issues.apache.org/jira/browse/SPARK-58789
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Shrirang Mhalgi
> Priority: Major
> Labels: pull-request-available
>
> Add a default method `additionalSparkProperties()` to the
> `CredentialProvider` interface that allows provider modules to declare Spark
> configuration properties needed when the provider is active.
> After successful credential acquisition, `UserCredentialManager.start()`
> iterates all discovered providers and applies their declared properties to
> `SparkConf` – only if the user has not already set them explicitly. This
> enables zero-config executor-side wiring (e.g., S3A credentials provider)
> without placing vendor-specific logic in core.
> h4. Motivation:
> - PR #57998 (SparkOidcAwsCredentialsProvider) originally placed auto-config
> logic in CoarseGrainedSchedulerBackend, which was a layer violation (core
> referencing an optional module's class)
> - This PR moves the mechanism to a generic SPI where each provider module
> declares its own properties
> h4. Changes:
> - CredentialProvider.java: new default method `additionalSparkProperties()`
> (returns Map<String, String>, empty by default)
> - CredentialProviderLoader.java: new `discoverAllProviders()` for querying
> provider metadata
> - UserCredentialManager.scala: apply provider properties after start()
> - AwsStsCredentialProvider.java: override returning S3A executor provider
> mapping
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
