Johannes Ehls created SPARK-62083:
-------------------------------------
Summary: Single-pass resolver resolves a function argument again
when its rewrite uses the argument twice
Key: SPARK-62083
URL: https://issues.apache.org/jira/browse/SPARK-62083
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.5.0, 5.0.0
Reporter: Johannes Ehls
The single-pass resolver resolves the arguments of a function call first and
then lets the function builder produce a replacement expression, which it
resolves as a subtree. To avoid resolving the already resolved arguments again,
{{ProducesUnresolvedSubtree}} marks them with the
{{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}}
returns a tagged node as is.
{{ExpressionResolver}} removes the tag on the first visit. A replacement that
references an argument more than once therefore resolves the second occurrence
again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}} (by default
it is false, and the rewrite then shares the argument through a common
expression instead of duplicating it):
{code:sql}
SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
{code}
{{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the
scalar subquery {{a}} is visited twice and its already resolved plan is
resolved a second time. The same applies to {{between}} and other rewrites that
duplicate an argument. The query still returns the right result, but the second
resolution is wasted work and breaks the single-pass invariant that every node
is resolved once.
Keeping the boundaries in tags has more problems:
* {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}), so
a boundary can end up on a node that is not a boundary.
* Setting tags mutates nodes that may be shared.
* For a named argument the tag is set on the {{NamedArgumentExpression}}
wrapper, which argument rearrangement replaces with its value, so the boundary
is lost.
Proposal: track the boundaries in resolver state instead of tags. A
{{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each
boundary node, by identity, to the number of open scopes. The
{{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced
subtree is resolved and remove them in a finally, and
{{ExpressionResolver.resolve}} only checks them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]