cloud-fan commented on code in PR #58045:
URL: https://github.com/apache/spark/pull/58045#discussion_r3805954200
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -98,6 +98,70 @@ object RewriteWithExpression extends Rule[LogicalPlan] {
}
}
+ /**
+ * Whether pre-evaluating this common expression in a conditional branch is
safe, where inlining
+ * is otherwise preferred because the branch may not be evaluated at all.
+ *
+ * It is safe only when the expression cannot raise. Pre-evaluating happens
for every row,
+ * including the rows whose branch is not taken, so an expression that
raises would turn a wrong
+ * result into a spurious error. That rules out more than it may seem:
`randstr(-1, 0)` raises on
+ * its constant length and `reflect(...)` raises on constant arguments, so
neither a
+ * nondeterministic root nor foldable children are enough. The generators
below produce a value
+ * from a seed or from the task context, evaluate no argument beyond a
foldable seed, and so have
+ * nothing to raise on. Any other nondeterministic expression, say `rand() /
col`, keeps the
+ * existing inlining and its existing wrong result.
+ *
+ * Only nondeterministic expressions are worth pre-evaluating here at all:
inlining a
+ * deterministic one repeats the work but returns the same value, so the
branch keeps deciding
+ * whether it runs.
+ */
+ private def canPreEvaluateInBranch(child: Expression): Boolean = {
+ val cannotRaise = child match {
+ // `rand`/`randn`: the seed is their only child and the analyzer rejects
a non-foldable one
+ // with `SEED_EXPRESSION_IS_UNFOLDABLE`, so there is no row data left to
raise on.
+ case _: NondeterministicUnaryRDG => true
+ // `uuid`, `monotonically_increasing_id`, `spark_partition_id`,
`input_file_name` and the
+ // input-file-block pair: no children to evaluate at all.
+ case _: LeafExpression => true
+ case _ => false
+ }
+ !child.deterministic && cannotRaise
+ }
+
+ /**
+ * Adds `child` to the input plan that supplies its references as a
pre-evaluated column, and
Review Comment:
**Nit:**
```suggestion
* Adds `child` as a pre-evaluated column to the input plan that supplies
`child`'s references, and
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
