Max Gekk created SPARK-60315:
--------------------------------
Summary: Filter raises an error with whole-stage codegen off but
not on when an earlier conjunct is null
Key: SPARK-60315
URL: https://issues.apache.org/jira/browse/SPARK-60315
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
The same query raises an error with whole-stage codegen off and returns rows
with it on. It happens when a filter's condition is a conjunction, an earlier
conjunct is NULL on a row, and a later conjunct raises on that row.
h3. Reproducer
Default (ANSI) mode, a Parquet table, local mode:
{code:sql}
CREATE TABLE t USING parquet AS
SELECT * FROM VALUES (NULL, 1, 2147483647), (0, 1, 0) AS v(a, b, y);
SET spark.sql.codegen.wholeStage=true;
SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
-- 0
SET spark.sql.codegen.wholeStage=false;
SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
-- [ARITHMETIC_OVERFLOW] overflow. Use 'try_add' to tolerate overflow and
return NULL instead. ... SQLSTATE: 22003
{code}
The same happens in legacy mode with year-month interval arithmetic, which
raises on overflow in every mode:
{code:sql}
SET spark.sql.ansi.enabled=false;
CREATE TABLE t2 USING parquet AS
SELECT * FROM VALUES (NULL, 1, 178956970), (0, 1, 0) AS v(a, b, y);
SELECT a FROM t2 WHERE NOT (a > 0 AND b > 0) AND make_ym_interval(y, 0) +
INTERVAL '1' YEAR > INTERVAL '0' YEAR;
-- codegen on: 0
-- codegen off: [INTERVAL_ARITHMETIC_OVERFLOW.WITH_SUGGESTION] Integer overflow
while operating with intervals. ... SQLSTATE: 22015
{code}
On the first row, {{NOT (a > 0 AND b > 0)}} is {{NOT (NULL AND TRUE)}}, which
is NULL, and the second conjunct overflows. The second row keeps the Parquet
row group from being skipped by its statistics; it passes both conjuncts on
both paths. {{NOT (a > 0 AND b > 0)}} is written so that the optimizer infers
no {{isnotnull}} filter that would drop the first row earlier on both paths.
Checked on master at 5d17442dec4 (8 October 2026) with {{bin/spark-sql --master
local[1]}}.
h3. Cause
The two paths evaluate the conjunction differently:
* *Whole-stage codegen* ({{FilterExec.generatePredicateCode}}) generates each
top-level conjunct separately and skips the row as soon as one is false *or
null* ({{if (isNull || !value) continue;}}). A filter keeps only rows whose
condition is true, so the remaining conjuncts are never evaluated on that row.
* *Interpreted* ({{FilterExec.doExecute}} without codegen, via
{{Predicate.create}} over the whole condition) evaluates the {{And}} tree with
{{And.eval}}, which short-circuits only on {{false}}. A null left side still
evaluates the right side, because {{NULL AND FALSE}} is false and {{NULL AND
TRUE}} is null, and that evaluation raises.
Both are valid evaluations of {{AND}} in isolation. Inside a filter, though,
false and null both drop the row, so the distinction the interpreted path
computes the right side for is never used.
h3. Expected
The query should behave the same with whole-stage codegen on and off.
Whole-stage codegen is the default, and its behaviour (no error) is the one
users see, so the interpreted filter should match it: evaluate the top-level
conjuncts in the same order and stop at the first one that is false or null.
The same applies to other interpreted evaluations of a filter condition that
codegen splits into conjuncts.
Found by differential testing of random queries against Spark's own row engine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
