[
https://issues.apache.org/jira/browse/SPARK-59639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59639:
-----------------------------------
Labels: pull-request-available (was: )
> Tighten nullability of Divide, Remainder, IntegralDivide, and Pmod under ANSI
> mode
> ----------------------------------------------------------------------------------
>
> Key: SPARK-59639
> URL: https://issues.apache.org/jira/browse/SPARK-59639
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Summary
> {{{}Divide{}}}, {{{}Remainder{}}}, {{IntegralDivide}} (which share the
> {{DivModLike}} trait) and {{Pmod}}
> hardcode {{{}override def nullable: Boolean = true{}}}. Under ANSI mode (the
> default), divide-by-zero and integral overflow *throw* instead of returning
> {{{}null{}}}, so these operators can only be {{{}null }}when one of their
> inputs is \{{{}null{}}}. The unconditional {{nullable = true}} is therefore
> over-broad under ANSI.
> h2. Why it matters
> Whole-stage and expression codegen represent SQL {{NULL}} with boolean
> {{isNull}} flags. A {{nullable = true}} child forces the framework to
> materialize an {{isNull}} variable and makes every parent expression emit a
> null-guard branch on it ({{{}CodegenContext.nullSafeExec{}}}). Because
> nullability propagates, one spuriously-nullable {{%}} cascades into the
> comparisons, {{{}CASE WHEN{}}}'s and predicates built on top of it,
> generating dead {{if (!isNull)}} branches that can never be taken.
> For example, {{(id % 2) = 0}} used inside a {{CASE WHEN}} generated:
> {code:java}
> boolean project_isNull_5 = false; // id % 2 -- never set to true
> if (!project_isNull_5) { ... } // dead guard
> ...
> if (!project_isNull_4 && project_value_4) // dead: !isNull_4 is always true
> here
> {code}
> h2. Root cause
> {{DivModLike.nullable}} and {{Pmod.nullable}} return {{true}}
> unconditionally. But {{eval}} returns {{null}} (beyond its children) only
> when {{{}!failOnError && isZero(divisor){}}}; integral overflow throws, and
> {{failOnError}} is {{{}evalMode == ANSI{}}}. So under ANSI the result is
> {{null}} iff an input is.
> h2. User-facing change
> No behavioral change. Runtime evaluation and code generation are unchanged –
> ANSI
> divide/remainder/pmod-by-zero and integral overflow still throw, and
> LEGACY/TRY still return {{{}null{}}}. Only the declared nullability tightens,
> and only under ANSI: an output column that is a division or modulo of
> non-nullable inputs may now be reported as non-nullable, which is more
> accurate.
> h2. Effect
> * Under ANSI, {{{}/{}}}, {{{}%{}}}, {{div}} and {{pmod}} over non-null
> inputs are now correctly
> non-nullable, eliminating the dead null-guard branches and their materialized
> flags at the source, and feeding more accurate nullability to the optimizer.
> * On representative codegen-heavy queries, total generated class bytecode
> dropped ~9.6% (fewer branches and StackMapTable frames; e.g. a filter with 30
> modulo predicates shrank ~36%). The reduction scales with how heavily a query
> uses these operators.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
