Francisco A. B. Sampaio created SPARK-59672:
-----------------------------------------------
Summary: SQL parameter binding FAILS with UNBOUND_SQL_PARAMETER
when a session extension injects a parser (4.1 regression)
Key: SPARK-59672
URL: https://issues.apache.org/jira/browse/SPARK-59672
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.3, 4.1.2, 4.1.1, 4.1.0
Reporter: Francisco A. B. Sampaio
Since 4.1.0, {{spark.sql(query, args)}} fails to bind parameters whenever a
session extension is configured via {{spark.sql.extensions}}. Both positional
({{?}}) and named ({{:name}}) parameters fail with {{UNBOUND_SQL_PARAMETER}}.
Unsetting {{spark.sql.extensions}} on the same build makes them work. 4.0.x is
unaffected.
Reproduced with two unrelated extensions, Delta Lake 4.1.0
({{io.delta.sql.DeltaSparkSessionExtension}}) and Apache Iceberg 1.11.0, so it
is not specific to either project. Both inject a parser through
{{SparkSessionExtensions.injectParser}}.
h3. Root cause
SPARK-53573 moved parameter handling into a parser pre-processor, and
{{SparkSession.sql}} now calls {{ParserInterface.parsePlanWithParameters}}.
That method has a default implementation that discards the parameters:
{code:scala}
def parsePlanWithParameters(sqlText: String, parameterContext:
ParameterContext): LogicalPlan = {
// Default implementation falls back to regular parsePlan
// Concrete implementations can override this for parameter support
parsePlan(sqlText)
}
{code}
Only {{SparkSqlParser}} overrides it. A parser injected by an extension
inherits the default, so the values never reach the parser and the markers
arrive at the analyzer unbound.
h3. Reproduction (spark-shell, no Spark Connect involved)
{code:scala}
// spark-shell --conf
spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql("SELECT ? AS id", ArrayAny).collect()
// org.apache.spark.sql.AnalysisException: [UNBOUND_SQL_PARAMETER] Found the
unbound parameter: _7.
spark.sql("SELECT :id AS id", Map("id" -> 42)).collect()
// [UNBOUND_SQL_PARAMETER] Found the unbound parameter: id.
// same build, extension unset: both return [42]
{code}
h3. Results
||Spark||spark.sql.extensions||Parameter binding||
|4.1.0 - 4.1.3|Delta 4.1.0|fails|
|4.1.3|Iceberg 1.11.0|fails|
|4.1.3|unset|works|
|4.0.4|Delta 4.0.1|works|
|4.0.4|Iceberg 1.10.2|works|
Setting only {{spark.sql.catalog.spark_catalog}} (Delta's catalog, without the
extension) does not trigger it.
Also reproducible through Spark Connect, for both the {{SQL}} relation and the
{{SqlCommand}} command, using {{pos_arguments}}, {{named_arguments}} and the
deprecated {{pos_args}}.
h3. Workaround
Setting {{spark.sql.legacy.parameterSubstitution.constantsOnly=true}} restores
binding for classic {{spark.sql(query, args)}} and for the Connect
{{SqlCommand}} path, because the analyzer binds the markers instead. It does
not help the Connect {{SQL}} relation path, which never applies the legacy
fallback - that looks like a separate defect and I can file it separately.
h3. Impact
Parameterized SQL is the standard defence against SQL injection. On 4.1.x it
fails outright for anyone using Delta, Iceberg or any other session extension,
pushing users toward string interpolation.
A PR is ready: the default implementation binds the parameters during analysis
so parsers that only override {{parsePlan}} keep working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
