[
https://issues.apache.org/jira/browse/SPARK-60483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60483:
-----------------------------------
Labels: pull-request-available (was: )
> Add spark.sql.extensions.defaultList for platform default session extensions
> ----------------------------------------------------------------------------
>
> Key: SPARK-60483
> URL: https://issues.apache.org/jira/browse/SPARK-60483
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: John Zhuge
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem
> Platform or cluster operators often need a set of SparkSessionExtensions
> active for every application (for example a catalog integration, a parser
> extension, or an optimizer rule), while application owners add their own
> extensions.
> {{spark.sql.extensions}} is a single list. If the operator sets it in
> spark-defaults.conf and a user passes
> {{{}--conf spark.sql.extensions=com.example.MyExtensions{}}}, the user value
> replaces
> the operator value and the operator's extensions are silently dropped. Today
> the user must know about the operator list and copy it by hand.
> h3. Prior art in Spark
> Spark already solves this for other list-valued configs with an
> operator-facing
> default list that is prepended to the user value, via
> ConfigBuilder.withPrepended:
> * {{spark.plugins.defaultList}} prepended to {{spark.plugins}} (SPARK-29397,
> 3.0.0)
> * {{spark.driver.defaultJavaOptions}} / {{spark.executor.defaultJavaOptions}}
> prepended to the matching extraJavaOptions
> SPARK-35380 (3.2.0) added SparkSessionExtensionsProvider via ServiceLoader. It
> works well for libraries that self-register, but fits the operator case less
> well: activation depends on classpath contents rather than configuration, the
> active set is not visible in the application's conf, it cannot be disabled
> without changing the classpath, and a failing provider is logged and skipped.
> h3. Proposal
> Add a static SQL conf {{spark.sql.extensions.defaultList}} (default: empty).
> The effective value of {{spark.sql.extensions}} becomes the default list
> followed by the user value, by applying withPrepended to
> SPARK_SESSION_EXTENSIONS, as spark.plugins does.
> {code:java}
> # spark-defaults.conf (operator)
> spark.sql.extensions.defaultList
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> # application
> --conf spark.sql.extensions=com.example.MyExtensions
> # effective spark.sql.extensions
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.example.MyExtensions
> {code}
> Semantics:
> * Unset: behavior is unchanged.
> * Order: default-list entries are applied before user entries. Order matters
> because parser extensions chain and injected rules run in registration order.
> * ServiceLoader providers (SPARK-35380) and
> SparkSession.Builder.withExtensions
> keep their current behavior and position.
> * A user can still opt out of a default by overriding defaultList itself,
> which
> keeps the operator/user split explicit.
> h3. Open questions
> # Should the effective list be de-duplicated, preserving order, as
> PluginContainer does for plugins? Applying the same extension twice injects
> its rules and parser wrapper twice. Proposed: yes.
> # Naming: {{spark.sql.extensions.defaultList}} mirrors
> {{{}spark.plugins.defaultList{}}}. Open to alternatives.
> h3. Testing
> Unit tests (e.g. in SparkSessionExtensionSuite) for: default list plus user
> list in order; default only; user only; unset leaves behavior unchanged;
> duplicates handled per the decision above.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
