David Mollitor created SPARK-59639:
--------------------------------------
Summary: Tighten nullability of Divide, Remainder, IntegralDivide,
and Pmod under ANSI mode
Key: SPARK-59639
URL: https://issues.apache.org/jira/browse/SPARK-59639
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
{{{}Divide{}}}, {{{}Remainder{}}}, {{IntegralDivide}} (which share the
{{DivModLike}} trait) and {{Pmod}}
hardcode {{{}override def nullable: Boolean = true{}}}. Under ANSI mode (the
default), divide-by-zero and integral overflow *throw* instead of returning
{{{}null{}}}, so these operators can only be {{{}null }}when one of their
inputs is \{{{}null{}}}. The unconditional {{nullable = true}} is therefore
over-broad under ANSI.
h2. Why it matters
Whole-stage and expression codegen represent SQL {{NULL}} with boolean
{{isNull}} flags. A {{nullable = true}} child forces the framework to
materialize an {{isNull}} variable and makes every parent expression emit a
null-guard branch on it ({{{}CodegenContext.nullSafeExec{}}}). Because
nullability propagates, one spuriously-nullable {{%}} cascades into the
comparisons, {{{}CASE WHEN{}}}'s and predicates built on top of it, generating
dead {{if (!isNull)}} branches that can never be taken.
For example, {{(id % 2) = 0}} used inside a {{CASE WHEN}} generated:
{code:java}
boolean project_isNull_5 = false; // id % 2 -- never set to true
if (!project_isNull_5) { ... } // dead guard
...
if (!project_isNull_4 && project_value_4) // dead: !isNull_4 is always true
here
{code}
h2. Root cause
{{DivModLike.nullable}} and {{Pmod.nullable}} return {{true}} unconditionally.
But {{eval}} returns {{null}} (beyond its children) only when {{{}!failOnError
&& isZero(divisor){}}}; integral overflow throws, and {{failOnError}} is
{{{}evalMode == ANSI{}}}. So under ANSI the result is {{null}} iff an input is.
h2. User-facing change
No behavioral change. Runtime evaluation and code generation are unchanged –
ANSI
divide/remainder/pmod-by-zero and integral overflow still throw, and LEGACY/TRY
still return {{{}null{}}}. Only the declared nullability tightens, and only
under ANSI: an output column that is a division or modulo of non-nullable
inputs may now be reported as non-nullable, which is more accurate.
h2. Effect
* Under ANSI, {{{}/{}}}, {{{}%{}}}, {{div}} and {{pmod}} over non-null inputs
are now correctly
non-nullable, eliminating the dead null-guard branches and their materialized
flags at the source, and feeding more accurate nullability to the optimizer.
* On representative codegen-heavy queries, total generated class bytecode
dropped ~9.6% (fewer branches and StackMapTable frames; e.g. a filter with 30
modulo predicates shrank ~36%). The reduction scales with how heavily a query
uses these operators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
