[ 
https://issues.apache.org/jira/browse/SPARK-62083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johannes Ehls updated SPARK-62083:
----------------------------------
    Component/s: Tests
    Description: 
Add golden test cases to scalar-subquery-select.sql for functions that are 
rewritten into an expression that references an argument more than once: 
{{nullif}}, {{between}} and {{decode}}. The arguments are scalar subqueries 
(also correlated), an aggregate, a window function and a lambda variable, with 
{{spark.sql.alwaysInlineCommonExpr}} on and off.

With {{spark.sql.alwaysInlineCommonExpr=true}}, {{nullif(a, b)}} is rewritten 
to {{NullIfResult(EqualTo(a, b), a)}}, so the same resolved argument occurs 
twice in the replacement, and the single-pass resolver can resolve it a second 
time. The tests pin the results of these shapes so that changes to how the 
analyzer handles duplicated arguments are covered by golden files.

  was:
The single-pass resolver resolves the arguments of a function call first and 
then lets the function builder produce a replacement expression, which it 
resolves as a subtree. To avoid resolving the already resolved arguments again, 
{{ProducesUnresolvedSubtree}} marks them with the 
{{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}} 
returns a tagged node as is.

{{ExpressionResolver}} removes the tag on the first visit. A replacement that 
references an argument more than once therefore resolves the second occurrence 
again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}} (by default 
it is false, and the rewrite then shares the argument through a common 
expression instead of duplicating it):

{code:sql}
SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
{code}

{{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the 
scalar subquery {{a}} is visited twice and its already resolved plan is 
resolved a second time. The same applies to {{between}} and other rewrites that 
duplicate an argument. The query still returns the right result, but the second 
resolution is wasted work and breaks the single-pass invariant that every node 
is resolved once.

Keeping the boundaries in tags has more problems:
* {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}), so 
a boundary can end up on a node that is not a boundary.
* Setting tags mutates nodes that may be shared.
* For a named argument the tag is set on the {{NamedArgumentExpression}} 
wrapper, which argument rearrangement replaces with its value, so the boundary 
is lost.

Proposal: track the boundaries in resolver state instead of tags. A 
{{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each 
boundary node, by identity, to the number of open scopes. The 
{{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced 
subtree is resolved and remove them in a finally, and 
{{ExpressionResolver.resolve}} only checks them.


     Issue Type: Test  (was: Bug)
        Summary: Add golden tests for functions whose rewrite uses an argument 
twice  (was: Single-pass resolver resolves a function argument again when its 
rewrite uses the argument twice)

> Add golden tests for functions whose rewrite uses an argument twice
> -------------------------------------------------------------------
>
>                 Key: SPARK-62083
>                 URL: https://issues.apache.org/jira/browse/SPARK-62083
>             Project: Spark
>          Issue Type: Test
>          Components: SQL, Tests
>    Affects Versions: 5.0.0, 4.5.0
>            Reporter: Johannes Ehls
>            Priority: Minor
>              Labels: pull-request-available
>
> Add golden test cases to scalar-subquery-select.sql for functions that are 
> rewritten into an expression that references an argument more than once: 
> {{nullif}}, {{between}} and {{decode}}. The arguments are scalar subqueries 
> (also correlated), an aggregate, a window function and a lambda variable, 
> with {{spark.sql.alwaysInlineCommonExpr}} on and off.
> With {{spark.sql.alwaysInlineCommonExpr=true}}, {{nullif(a, b)}} is rewritten 
> to {{NullIfResult(EqualTo(a, b), a)}}, so the same resolved argument occurs 
> twice in the replacement, and the single-pass resolver can resolve it a 
> second time. The tests pin the results of these shapes so that changes to how 
> the analyzer handles duplicated arguments are covered by golden files.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to