[ 
https://issues.apache.org/jira/browse/SPARK-62083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62083:
-----------------------------------
    Labels: pull-request-available  (was: )

> Single-pass resolver resolves a function argument again when its rewrite uses 
> the argument twice
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62083
>                 URL: https://issues.apache.org/jira/browse/SPARK-62083
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0, 4.5.0
>            Reporter: Johannes Ehls
>            Priority: Minor
>              Labels: pull-request-available
>
> The single-pass resolver resolves the arguments of a function call first and 
> then lets the function builder produce a replacement expression, which it 
> resolves as a subtree. To avoid resolving the already resolved arguments 
> again, {{ProducesUnresolvedSubtree}} marks them with the 
> {{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}} 
> returns a tagged node as is.
> {{ExpressionResolver}} removes the tag on the first visit. A replacement that 
> references an argument more than once therefore resolves the second 
> occurrence again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}} 
> (by default it is false, and the rewrite then shares the argument through a 
> common expression instead of duplicating it):
> {code:sql}
> SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
> {code}
> {{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the 
> scalar subquery {{a}} is visited twice and its already resolved plan is 
> resolved a second time. The same applies to {{between}} and other rewrites 
> that duplicate an argument. The query still returns the right result, but the 
> second resolution is wasted work and breaks the single-pass invariant that 
> every node is resolved once.
> Keeping the boundaries in tags has more problems:
> * {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}), 
> so a boundary can end up on a node that is not a boundary.
> * Setting tags mutates nodes that may be shared.
> * For a named argument the tag is set on the {{NamedArgumentExpression}} 
> wrapper, which argument rearrangement replaces with its value, so the 
> boundary is lost.
> Proposal: track the boundaries in resolver state instead of tags. A 
> {{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each 
> boundary node, by identity, to the number of open scopes. The 
> {{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced 
> subtree is resolved and remove them in a finally, and 
> {{ExpressionResolver.resolve}} only checks them.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to