[
https://issues.apache.org/jira/browse/SPARK-60749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60749:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Do not reorder associative operators across different evaluation modes
> ----------------------------------------------------------------------
>
> Key: SPARK-60749
> URL: https://issues.apache.org/jira/browse/SPARK-60749
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Joel Robin
> Priority: Major
> Labels: correctness, pull-request-available
>
> {{ReorderAssociativeOperator}} flattens nested integral {{Add}} /
> {{Multiply}} chains to fold their literals, but it ignores each operator's
> evaluation mode and rebuilds the whole chain in the root operator's mode.
> Since SPARK-40222, numeric {{try_add}} and {{try_multiply}} are replaced by
> TRY-mode {{Add}} / {{Multiply}}, so combining them with regular arithmetic
> over non-foldable inputs changes their result:
> {code:sql}
> CREATE TABLE t1 (x INT, y INT) USING parquet;
> INSERT INTO t1 VALUES (2147483647, 1);
> SELECT try_add(x, y) + 1 FROM t1; -- ARITHMETIC_OVERFLOW, expected NULL
> SELECT try_multiply(x, 2) * 2 FROM t1; -- ARITHMETIC_OVERFLOW, expected NULL
> SELECT try_add(x + y, 1) FROM t1; -- NULL, expected ARITHMETIC_OVERFLOW
> from x + y
> {code}
> For the first query the analyzed plan is {{(try_add(x, y) + 1)}}, but the
> optimized plan is {{((x + y) + 1)}} with the inner addition evaluated in ANSI
> mode. With the rule excluded
> ({{spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator}})
> the query returns NULL. The third query is the reverse case: the ANSI {{x +
> y}} is rebuilt in TRY mode, so its overflow becomes NULL, although
> SPARK-40222 requires errors from the children of try_* functions to propagate.
> Literal inputs are not affected, because ConstantFolding evaluates
> {{try_add}} before this rule runs. The rule already has a TODO for this: "do
> not reorder consecutive Adds or Multiplys with different failOnError flags".
> Proposed fix: only flatten operators whose evaluation context equals the
> root's, and keep an operator in a different mode as a single operand. PR:
> https://github.com/apache/spark/pull/59954
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
