LuciferYang opened a new pull request, #59209:
URL: https://github.com/apache/spark/pull/59209
### What changes were proposed in this pull request?
`RewriteWithExpression` pre-evaluates a common expression's definition in a
`Project` over the first child whose output covers the definition's references.
A default value references no column, so for a `CreateVariable` the rule put
that `Project` over the first `ResolvedIdentifier`, which names a variable and
is not an input. `V2CommandStrategy` plans a `CreateVariable` only when every
child is a `ResolvedIdentifier`, so planning failed.
This adds a `CreateVariable` case to the rule. It inlines the definitions
that gain nothing from memoizing, as it does for a `With` in a conditional
branch, and leaves the rest in the default. `CreateVariableExec` evaluates the
default once on the driver, and a `With` can be evaluated since SPARK-58818, so
a kept definition is evaluated once.
### Why are the changes needed?
```sql
DECLARE OR REPLACE VARIABLE v1 DOUBLE DEFAULT nullif(rand(), 0.5);
```
fails with
```
[INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error.
... SQLSTATE: XX000
Caused by: java.lang.AssertionError: assertion failed: No plan for
CreateVariable default(if ((_common_expr_0#0 = 0.5)) null else
_common_expr_0#0) ...
+- Project [rand(...) AS _common_expr_0#0]
+- ResolvedIdentifier ..., session.v1
```
Any NULLIF or BETWEEN default whose operand is not cheap fails the same way,
a deterministic UDF included. A cheap operand, such as `nullif(1, 3)`, is
inlined and works.
### Does this PR introduce _any_ user-facing change?
Yes. A DECLARE VARIABLE like the one above now creates the variable instead
of failing with INTERNAL_ERROR.
### How was this patch tested?
- `RewriteWithExpressionSuite`: a `CreateVariable` whose default holds a
`With` worth memoizing comes out of the rule unchanged, and one whose
definition gains nothing is inlined.
- `ColumnExpressionSuite`: a NULLIF default and a BETWEEN default over a
counting UDF each create the variable, call the UDF once, and read back the
expected value.
Both fail without the change to `RewriteWithExpression`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
