Yang Jie created SPARK-59955:
--------------------------------
Summary: Replace the stale TODOs in RewriteWithExpression's
fallback for a definition no single child can evaluate
Key: SPARK-59955
URL: https://issues.apache.org/jira/browse/SPARK-59955
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
When a common expression's definition reads columns from more than one child,
as a join condition reading both sides does, RewriteWithExpression cannot
pre-evaluate it in any child and inlines it into every reference. That branch
carries two TODOs, and neither describes something that can still happen.
1. "things can go wrong if the common expression is nondeterministic". Only an
operator with more than one child reaches the branch. Analysis rejects a
nondeterministic join condition (INVALID_NON_DETERMINISTIC_EXPRESSIONS), a
nondeterministic LATERAL join condition
(LATERAL_JOIN_CONDITION_NON_DETERMINISTIC) and a nondeterministic MERGE
condition. NearestByJoin allows a nondeterministic ranking expression, but
RewriteNearestByJoin replaces the node in the "Finish Analysis" batch, before
the "Rewrite With expression" batch runs.
2. "we should calculate the ref count and also inline the common expression if
it's ref count is 1". SPARK-58818 added canSubstitute ahead of this branch, and
it already inlines a definition referenced at most once.
The first TODO should become a statement of the invariant, checked by an
assertion when testing, and the second can be removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
