Tim Meehan created SPARK-58782:
----------------------------------
Summary: DSv2 pushdown serializes a null literal as the string
`'null'` instead of SQL `NULL`, returning wrong results
Key: SPARK-58782
URL: https://issues.apache.org/jira/browse/SPARK-58782
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Environment: Environment independent
Reporter: Tim Meehan
Problem When a null literal appears in a pushed-down DSv2 expression (JDBC
aggregate/predicate pushdown), it is serialized into the pushed SQL as the
4-character string literal {{'null'}} instead of the SQL keyword {{{}NULL{}}}.
The external engine then executes a different predicate/grouping key than
intended and silently returns wrong results. Because a connector
{{{}Expression{}}}'s {{describe()}} defaults to {{{}toString(){}}}, the same
wrong text also appears in {{EXPLAIN}} output.
Reproduction (H2, self-contained — no external DB):
scala
// table null_literal(s TEXT), rows: 'keep', '', NULL;
pushDownAggregate=truesql("SELECT NULLIF(s, '') AS g, COUNT(*) FROM
h2.test.null_literal GROUP BY g")
{{NULLIF(s,'')}} rewrites to {{{}If(s='', null, s){}}}, which is pushed as a
{{CASE_WHEN}} grouping key. The generated SQL is:
sql
SELECT CASE WHEN "S" = '' THEN 'null' ELSE "S" END, COUNT(*) FROM ... GROUP BY
CASE WHEN "S" = '' THEN 'null' ELSE "S" END
The {{''}} row is bucketed under the string {{"null"}} instead of collapsing
into the real {{NULL}} group — 3 result rows instead of 2.
Root cause {{LiteralValue.toString}} matches on {{dataType}} before checking
the value, so a null {{StringType}} value flows into the {{'…'}} string branch
and renders as {{{}'null'{}}}. (A null {{BinaryType}} value hits an {{assert}}
and throws; numeric/boolean nulls render as an unquoted {{{}null{}}}.)
Proposed fix In {{{}LiteralValue.toString{}}}, return the bare keyword {{NULL}}
for any null value (guard as the first {{{}case{}}}). A null literal is SQL
{{NULL}} regardless of type, so this is dialect-agnostic and fixes every
consumer (JDBC dialects, {{{}ToStringSQLBuilder{}}}/EXPLAIN) at once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
