John Zhuge created SPARK-60483:
----------------------------------
Summary: Add spark.sql.extensions.defaultList for platform default
session extensions
Key: SPARK-60483
URL: https://issues.apache.org/jira/browse/SPARK-60483
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: John Zhuge
h3. Problem
Platform or cluster operators often need a set of SparkSessionExtensions active
for every application (for example a catalog integration, a parser extension,
or an optimizer rule), while application owners add their own extensions.
{{spark.sql.extensions}} is a single list. If the operator sets it in
spark-defaults.conf and a user passes
{{{}--conf spark.sql.extensions=com.example.MyExtensions{}}}, the user value
replaces
the operator value and the operator's extensions are silently dropped. Today
the user must know about the operator list and copy it by hand.
h3. Prior art in Spark
Spark already solves this for other list-valued configs with an operator-facing
default list that is prepended to the user value, via
ConfigBuilder.withPrepended:
* {{spark.plugins.defaultList}} prepended to {{spark.plugins}} (SPARK-29397,
3.0.0)
* {{spark.driver.defaultJavaOptions}} / {{spark.executor.defaultJavaOptions}}
prepended to the matching extraJavaOptions
SPARK-35380 (3.2.0) added SparkSessionExtensionsProvider via ServiceLoader. It
works well for libraries that self-register, but fits the operator case less
well: activation depends on classpath contents rather than configuration, the
active set is not visible in the application's conf, it cannot be disabled
without changing the classpath, and a failing provider is logged and skipped.
h3. Proposal
Add a static SQL conf {{spark.sql.extensions.defaultList}} (default: empty).
The effective value of {{spark.sql.extensions}} becomes the default list
followed by the user value, by applying withPrepended to
SPARK_SESSION_EXTENSIONS, as spark.plugins does.
{code:java}
# spark-defaults.conf (operator)
spark.sql.extensions.defaultList
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
# application
--conf spark.sql.extensions=com.example.MyExtensions
# effective spark.sql.extensions
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,com.example.MyExtensions
{code}
Semantics:
* Unset: behavior is unchanged.
* Order: default-list entries are applied before user entries. Order matters
because parser extensions chain and injected rules run in registration order.
* ServiceLoader providers (SPARK-35380) and SparkSession.Builder.withExtensions
keep their current behavior and position.
* A user can still opt out of a default by overriding defaultList itself, which
keeps the operator/user split explicit.
h3. Open questions
# Should the effective list be de-duplicated, preserving order, as
PluginContainer does for plugins? Applying the same extension twice injects
its rules and parser wrapper twice. Proposed: yes.
# Naming: {{spark.sql.extensions.defaultList}} mirrors
{{{}spark.plugins.defaultList{}}}. Open to alternatives.
h3. Testing
Unit tests (e.g. in SparkSessionExtensionSuite) for: default list plus user
list in order; default only; user only; unset leaves behavior unchanged;
duplicates handled per the decision above.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
