Yang Jie created SPARK-59295:
--------------------------------
Summary: Emit a common expression's definition once per scope
under whole-stage codegen
Key: SPARK-59295
URL: https://issues.apache.org/jira/browse/SPARK-59295
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
`With` became evaluable in SPARK-58818, so a common expression that a
conditional branch reads more than once is memoized rather than substituted at
every reference. `CommonExprSlots.fill` puts the definition's code in a method
where it can, which leaves a reference's own code a single call. That is only
possible where the definition reads the input row: a whole-stage `Project` or
`Filter` hands an expression its input as local variables (`currentVars` set),
and neither `fill` nor `Expression.reduceCodeSize` can split under that
condition, so the body is pasted once per reference.
For a nested `With` the paste count doubles per level. Measured on a
`CodegenContext` with `currentVars` set, counting the innermost definition's
body: 2, 4, 8, ... 256 at depths 1 to 8. Where the input arrives as a row it is
2 at any depth.
Passing the `currentVars` values a definition reads into the method as
parameters, the way `subexpressionEliminationForWholeStageCodegen` does, would
fix it, but needs a decision first: `getLocalInputVariableValues` hoists an
input variable that is not evaluated yet to before the call, which for a
reference behind a branch means evaluating it on rows that never reach the
reference.
The decision point in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala`
carries a TODO pointing here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
