[
https://issues.apache.org/jira/browse/SPARK-61467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-61467:
-----------------------------
Description:
When a rule replaces a node, TreeNode.transformDown/transformUp call
copyTagsFrom on the expression the rule returns, and
SupportQueryContext.copyTagsFrom sets its mutable queryContext field in place.
That instance can be shared with other plans: a temp view's stored analyzed
plan, or a parent Dataset, since QueryExecution clones the plan nodes but not
their expressions. So the error of one query can report the context of another.
With ANSI on:
{code:scala}
sql("SELECT CAST(CAST(id - 2147483648 AS INT) AS INTERVAL MONTH) / -1 AS d FROM
range(1)")
.createOrReplaceTempView("v")
sql("SELECT CAST(d AS INTERVAL YEAR TO MONTH) FROM v").collect() // (1)
sql("SELECT d FROM v").collect() // (2)
{code}
In (1), SimplifyCasts returns the view's DivideYMInterval instance in place of
the same-type cast, which copies the cast's context into it. (2) then fails
with the fragment CAST(d AS INTERVAL YEAR TO MONTH) and the SQL text of (1),
with whole-stage codegen on and off. The same happens with a Dataset:
base.select($"d".cast("interval year to month")) changes the context that a
later base.collect() reports.
This affects every SupportQueryContext expression, such as ANSI Add; the
interval division joined them in SPARK-61352. copyTagsFrom should not change an
expression that other plans can share, e.g. by returning a copy with the new
context.
> SupportQueryContext.copyTagsFrom overwrites the query context of an
> expression shared with other plans
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61467
> URL: https://issues.apache.org/jira/browse/SPARK-61467
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
>
> When a rule replaces a node, TreeNode.transformDown/transformUp call
> copyTagsFrom on the expression the rule returns, and
> SupportQueryContext.copyTagsFrom sets its mutable queryContext field in
> place. That instance can be shared with other plans: a temp view's stored
> analyzed plan, or a parent Dataset, since QueryExecution clones the plan
> nodes but not their expressions. So the error of one query can report the
> context of another. With ANSI on:
> {code:scala}
> sql("SELECT CAST(CAST(id - 2147483648 AS INT) AS INTERVAL MONTH) / -1 AS d
> FROM range(1)")
> .createOrReplaceTempView("v")
> sql("SELECT CAST(d AS INTERVAL YEAR TO MONTH) FROM v").collect() // (1)
> sql("SELECT d FROM v").collect() // (2)
> {code}
> In (1), SimplifyCasts returns the view's DivideYMInterval instance in place
> of the same-type cast, which copies the cast's context into it. (2) then
> fails with the fragment CAST(d AS INTERVAL YEAR TO MONTH) and the SQL text of
> (1), with whole-stage codegen on and off. The same happens with a Dataset:
> base.select($"d".cast("interval year to month")) changes the context that a
> later base.collect() reports.
> This affects every SupportQueryContext expression, such as ANSI Add; the
> interval division joined them in SPARK-61352. copyTagsFrom should not change
> an expression that other plans can share, e.g. by returning a copy with the
> new context.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]