Joel Robin created SPARK-60749:
----------------------------------
Summary: Do not reorder associative operators across different
evaluation modes
Key: SPARK-60749
URL: https://issues.apache.org/jira/browse/SPARK-60749
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Joel Robin
{{ReorderAssociativeOperator}} flattens nested integral {{Add}} / {{Multiply}}
chains to fold their literals, but it ignores each operator's evaluation mode
and rebuilds the whole chain in the root operator's mode. Since SPARK-40222,
numeric {{try_add}} and {{try_multiply}} are replaced by TRY-mode {{Add}} /
{{Multiply}}, so combining them with regular arithmetic over non-foldable
inputs changes their result:
{code:sql}
CREATE TABLE t1 (x INT, y INT) USING parquet;
INSERT INTO t1 VALUES (2147483647, 1);
SELECT try_add(x, y) + 1 FROM t1; -- ARITHMETIC_OVERFLOW, expected NULL
SELECT try_multiply(x, 2) * 2 FROM t1; -- ARITHMETIC_OVERFLOW, expected NULL
SELECT try_add(x + y, 1) FROM t1; -- NULL, expected ARITHMETIC_OVERFLOW
from x + y
{code}
For the first query the analyzed plan is {{(try_add(x, y) + 1)}}, but the
optimized plan is {{((x + y) + 1)}} with the inner addition evaluated in ANSI
mode. With the rule excluded
({{spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator}})
the query returns NULL. The third query is the reverse case: the ANSI {{x +
y}} is rebuilt in TRY mode, so its overflow becomes NULL, although SPARK-40222
requires errors from the children of try_* functions to propagate.
Literal inputs are not affected, because ConstantFolding evaluates {{try_add}}
before this rule runs. The rule already has a TODO for this: "do not reorder
consecutive Adds or Multiplys with different failOnError flags".
Proposed fix: only flatten operators whose evaluation context equals the
root's, and keep an operator in a different mode as a single operand. PR:
https://github.com/apache/spark/pull/59954
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
