[
https://issues.apache.org/jira/browse/SPARK-62083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62083:
-----------------------------------
Labels: pull-request-available (was: )
> Single-pass resolver resolves a function argument again when its rewrite uses
> the argument twice
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-62083
> URL: https://issues.apache.org/jira/browse/SPARK-62083
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0, 4.5.0
> Reporter: Johannes Ehls
> Priority: Minor
> Labels: pull-request-available
>
> The single-pass resolver resolves the arguments of a function call first and
> then lets the function builder produce a replacement expression, which it
> resolves as a subtree. To avoid resolving the already resolved arguments
> again, {{ProducesUnresolvedSubtree}} marks them with the
> {{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}}
> returns a tagged node as is.
> {{ExpressionResolver}} removes the tag on the first visit. A replacement that
> references an argument more than once therefore resolves the second
> occurrence again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}}
> (by default it is false, and the rewrite then shares the argument through a
> common expression instead of duplicating it):
> {code:sql}
> SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
> {code}
> {{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the
> scalar subquery {{a}} is visited twice and its already resolved plan is
> resolved a second time. The same applies to {{between}} and other rewrites
> that duplicate an argument. The query still returns the right result, but the
> second resolution is wasted work and breaks the single-pass invariant that
> every node is resolved once.
> Keeping the boundaries in tags has more problems:
> * {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}),
> so a boundary can end up on a node that is not a boundary.
> * Setting tags mutates nodes that may be shared.
> * For a named argument the tag is set on the {{NamedArgumentExpression}}
> wrapper, which argument rearrangement replaces with its value, so the
> boundary is lost.
> Proposal: track the boundaries in resolver state instead of tags. A
> {{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each
> boundary node, by identity, to the number of open scopes. The
> {{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced
> subtree is resolved and remove them in a finally, and
> {{ExpressionResolver.resolve}} only checks them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]