Johannes Ehls created SPARK-62083:
-------------------------------------

             Summary: Single-pass resolver resolves a function argument again 
when its rewrite uses the argument twice
                 Key: SPARK-62083
                 URL: https://issues.apache.org/jira/browse/SPARK-62083
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.5.0, 5.0.0
            Reporter: Johannes Ehls


The single-pass resolver resolves the arguments of a function call first and 
then lets the function builder produce a replacement expression, which it 
resolves as a subtree. To avoid resolving the already resolved arguments again, 
{{ProducesUnresolvedSubtree}} marks them with the 
{{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}} 
returns a tagged node as is.

{{ExpressionResolver}} removes the tag on the first visit. A replacement that 
references an argument more than once therefore resolves the second occurrence 
again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}} (by default 
it is false, and the rewrite then shares the argument through a common 
expression instead of duplicating it):

{code:sql}
SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
{code}

{{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the 
scalar subquery {{a}} is visited twice and its already resolved plan is 
resolved a second time. The same applies to {{between}} and other rewrites that 
duplicate an argument. The query still returns the right result, but the second 
resolution is wasted work and breaks the single-pass invariant that every node 
is resolved once.

Keeping the boundaries in tags has more problems:
* {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}), so 
a boundary can end up on a node that is not a boundary.
* Setting tags mutates nodes that may be shared.
* For a named argument the tag is set on the {{NamedArgumentExpression}} 
wrapper, which argument rearrangement replaces with its value, so the boundary 
is lost.

Proposal: track the boundaries in resolver state instead of tags. A 
{{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each 
boundary node, by identity, to the number of open scopes. The 
{{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced 
subtree is resolved and remove them in a finally, and 
{{ExpressionResolver.resolve}} only checks them.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to