[
https://issues.apache.org/jira/browse/SPARK-62083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johannes Ehls updated SPARK-62083:
----------------------------------
Component/s: Tests
Description:
Add golden test cases to scalar-subquery-select.sql for functions that are
rewritten into an expression that references an argument more than once:
{{nullif}}, {{between}} and {{decode}}. The arguments are scalar subqueries
(also correlated), an aggregate, a window function and a lambda variable, with
{{spark.sql.alwaysInlineCommonExpr}} on and off.
With {{spark.sql.alwaysInlineCommonExpr=true}}, {{nullif(a, b)}} is rewritten
to {{NullIfResult(EqualTo(a, b), a)}}, so the same resolved argument occurs
twice in the replacement, and the single-pass resolver can resolve it a second
time. The tests pin the results of these shapes so that changes to how the
analyzer handles duplicated arguments are covered by golden files.
was:
The single-pass resolver resolves the arguments of a function call first and
then lets the function builder produce a replacement expression, which it
resolves as a subtree. To avoid resolving the already resolved arguments again,
{{ProducesUnresolvedSubtree}} marks them with the
{{SINGLE_PASS_SUBTREE_BOUNDARY}} tag, and {{ExpressionResolver.resolve}}
returns a tagged node as is.
{{ExpressionResolver}} removes the tag on the first visit. A replacement that
references an argument more than once therefore resolves the second occurrence
again. For example, with {{spark.sql.alwaysInlineCommonExpr=true}} (by default
it is false, and the rewrite then shares the argument through a common
expression instead of duplicating it):
{code:sql}
SELECT nullif((SELECT max(id) FROM range(1, 4)), 0)
{code}
{{nullif(a, b)}} is replaced by {{NullIfResult(EqualTo(a, b), a)}}, so the
scalar subquery {{a}} is visited twice and its already resolved plan is
resolved a second time. The same applies to {{between}} and other rewrites that
duplicate an argument. The query still returns the right result, but the second
resolution is wasted work and breaks the single-pass invariant that every node
is resolved once.
Keeping the boundaries in tags has more problems:
* {{TreeNode.transform}} copies tags onto rebuilt nodes ({{copyTagsFrom}}), so
a boundary can end up on a node that is not a boundary.
* Setting tags mutates nodes that may be shared.
* For a named argument the tag is set on the {{NamedArgumentExpression}}
wrapper, which argument rearrangement replaces with its value, so the boundary
is lost.
Proposal: track the boundaries in resolver state instead of tags. A
{{SubtreeBoundaryScopes}} object owned by {{ExpressionResolver}} maps each
boundary node, by identity, to the number of open scopes. The
{{ProducesUnresolvedSubtree}} helpers add the boundaries while the produced
subtree is resolved and remove them in a finally, and
{{ExpressionResolver.resolve}} only checks them.
Issue Type: Test (was: Bug)
Summary: Add golden tests for functions whose rewrite uses an argument
twice (was: Single-pass resolver resolves a function argument again when its
rewrite uses the argument twice)
> Add golden tests for functions whose rewrite uses an argument twice
> -------------------------------------------------------------------
>
> Key: SPARK-62083
> URL: https://issues.apache.org/jira/browse/SPARK-62083
> Project: Spark
> Issue Type: Test
> Components: SQL, Tests
> Affects Versions: 5.0.0, 4.5.0
> Reporter: Johannes Ehls
> Priority: Minor
> Labels: pull-request-available
>
> Add golden test cases to scalar-subquery-select.sql for functions that are
> rewritten into an expression that references an argument more than once:
> {{nullif}}, {{between}} and {{decode}}. The arguments are scalar subqueries
> (also correlated), an aggregate, a window function and a lambda variable,
> with {{spark.sql.alwaysInlineCommonExpr}} on and off.
> With {{spark.sql.alwaysInlineCommonExpr=true}}, {{nullif(a, b)}} is rewritten
> to {{NullIfResult(EqualTo(a, b), a)}}, so the same resolved argument occurs
> twice in the replacement, and the single-pass resolver can resolve it a
> second time. The tests pin the results of these shapes so that changes to how
> the analyzer handles duplicated arguments are covered by golden files.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]