This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 103a0a77a70 [fix](expr opt) Preserve cast boundaries during arithmetic
simplification (#67733)
103a0a77a70 is described below
commit 103a0a77a7043ba16ea2a26235fbde6a3bae737e
Author: morrySnow <[email protected]>
AuthorDate: Fri Sep 11 14:41:14 2026 +0800
[fix](expr opt) Preserve cast boundaries during arithmetic simplification
(#67733)
## Problem
Arithmetic simplification can move a `CAST` or `TRY_CAST` from around an
arithmetic expression to each operand. That changes where type
conversion happens. With large exact integers converted to `DOUBLE`, the
rewritten expression loses precision before subtraction and returns the
wrong value.
For example, subtracting `9223372036854775800` from
`9223372036854775807` in `BIGINT` produces the exact value `7`.
Converting that result to `DOUBLE` must still produce `7`, but
converting both operands first rounds them to the same floating-point
value and produces `0`.
## Root cause
The arithmetic flattener recognized a cast containing addition,
subtraction, multiplication, or division and recursively propagated the
cast target type to every operand. This rewrite assumes that conversion
distributes over arithmetic, which is not generally true. Besides
floating-point rounding, the transformation can alter overflow,
conversion-error, and `TRY_CAST` null behavior.
## Reproduction
```sql
CREATE TABLE t (
id INT NOT NULL,
k BIGINT NOT NULL
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
INSERT INTO t VALUES
(1, 9223372036854775807),
(2, 9223372036854775806);
SELECT id,
k - 9223372036854775800 AS exact_delta,
TRY_CAST(k - 9223372036854775800 AS DOUBLE) + CAST(0 AS DOUBLE) AS
try_expr,
CAST(k - 9223372036854775800 AS DOUBLE) + CAST(0 AS DOUBLE) AS
cast_expr
FROM t
ORDER BY id;
```
Before this change, `exact_delta` is `7`/`6`, while both converted
expressions incorrectly return `0`/`0`.
## Fix
Treat explicit cast expressions as semantic boundaries and atomic
operands during arithmetic flattening. The simplifier can continue
optimizing arithmetic below and above a cast, but it no longer
distributes the conversion into the cast's operands.
This deliberately chooses correctness over the optimization enabled by
cross-cast flattening; no target-type whitelist is used because
conversion distribution is not safe across all values and operation
types.
The P0 run also exposed a regression-test fragility: a session `SET`
statement followed 167 lines of disabled Presto timestamp queries, so
the SQL splitter sent the comments and the `SET` together to the
dialect-conversion service. Move the `SET` before that disabled block so
it is converted and executed as a standalone statement. Query order and
expected result tags remain unchanged.
A separate chained-MTMV failure came from the task-wait helper
temporarily falling back to the preceding task after it had already
observed the new task. The branch is rebased onto current master, which
contains the framework correction that keeps waiting for the observed
task instead of latching onto the previous one.
The rebase also exposed a recently added IVM join-normalization test
that still called a removed rewrite-context constructor. Build the test
context through the supported CREATE-mode factory and set its full-key
flag explicitly, matching the production setup path and restoring FE
test-source compilation.
## Tests
- Added expression-rule coverage for both `CAST` and `TRY_CAST` around a
large-integer subtraction.
- `SimplifyArithmeticRuleTest`: 5 tests passed.
- Deployed the FE to a local sandbox. The reproduction returns `7`/`6`
for both converted expressions, and the physical expression keeps the
`BIGINT` subtraction inside the cast.
- Full Presto `TestOperators` suite with `doris-sql-convertor` 1.0.5:
passed.
- `SuiteMTMVTaskWaitTest`: 4 tests passed.
- `IvmNormalizeMTMVJoinTest`: 44 tests passed.
---
.../expression/rules/SimplifyArithmeticRule.java | 56 ++++------------------
.../expression/SimplifyArithmeticRuleTest.java | 13 +++++
.../sql/presto/scalar/timestamp/TestOperators.sql | 2 +-
3 files changed, 23 insertions(+), 48 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SimplifyArithmeticRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SimplifyArithmeticRule.java
index 6eea495e5cf..3076e36ec52 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SimplifyArithmeticRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SimplifyArithmeticRule.java
@@ -22,12 +22,10 @@ import
org.apache.doris.nereids.rules.expression.ExpressionPatternRuleFactory;
import org.apache.doris.nereids.rules.expression.ExpressionRuleType;
import org.apache.doris.nereids.trees.expressions.Add;
import org.apache.doris.nereids.trees.expressions.BinaryArithmetic;
-import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.Divide;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Multiply;
import org.apache.doris.nereids.trees.expressions.Subtract;
-import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.util.TypeCoercionUtils;
import org.apache.doris.nereids.util.TypeUtils;
import org.apache.doris.nereids.util.Utils;
@@ -138,16 +136,13 @@ public class SimplifyArithmeticRule implements
ExpressionPatternRuleFactory {
// isAddOrSub: true for extract only "+" or "-" sub expressions, false for
extract only "*" or "/" sub expressions
private static List<Operand> flatten(Expression expr, boolean isAddOrSub) {
List<Operand> result = Lists.newArrayList();
- doFlatten(true, expr, isAddOrSub, result, Optional.empty());
+ doFlatten(true, expr, isAddOrSub, result);
return result;
}
// flag: true for '+' or '*', false for '-' or '/'
// isAddOrSub: true for extract only "+" or "-" sub expressions, false for
extract only "*" or "/" sub expressions
- private static void doFlatten(boolean flag, Expression expr, boolean
isAddOrSub, List<Operand> result,
- Optional<DataType> castType) {
- // cast (a * 10 as double) * (cast 20 as double)
- // => cast(a as double) * (cast 10 as double) * (cast 20 as double)
+ private static void doFlatten(boolean flag, Expression expr, boolean
isAddOrSub, List<Operand> result) {
BinaryArithmetic arithmetic = null;
Predicate<Expression> isPositiveArithmetic = isAddOrSub
? TypeUtils::isAdd : TypeUtils::isMultiply;
@@ -156,52 +151,20 @@ public class SimplifyArithmeticRule implements
ExpressionPatternRuleFactory {
Predicate<Expression> isPosNegArithmetic =
isPositiveArithmetic.or(isNegativeArithmetic);
if (isPosNegArithmetic.test(expr)) {
arithmetic = (BinaryArithmetic) expr;
- } else if (expr instanceof Cast && hasConstantOperand(expr,
isAddOrSub)) {
- Cast cast = (Cast) expr;
- if (isPosNegArithmetic.test(cast.child())) {
- arithmetic = (BinaryArithmetic) cast.child();
- castType = Optional.of(cast.getDataType());
- }
}
if (arithmetic != null) {
- doFlatten(flag, arithmetic.left(), isAddOrSub, result, castType);
+ doFlatten(flag, arithmetic.left(), isAddOrSub, result);
if (isNegativeArithmetic.test(arithmetic) && !flag) {
- doFlatten(true, arithmetic.right(), isAddOrSub, result,
castType);
+ doFlatten(true, arithmetic.right(), isAddOrSub, result);
} else if (isPositiveArithmetic.test(arithmetic) && !flag) {
- doFlatten(false, arithmetic.right(), isAddOrSub, result,
castType);
+ doFlatten(false, arithmetic.right(), isAddOrSub, result);
} else {
- doFlatten(!isNegativeArithmetic.test(arithmetic),
arithmetic.right(), isAddOrSub, result, castType);
+ doFlatten(!isNegativeArithmetic.test(arithmetic),
arithmetic.right(), isAddOrSub, result);
}
} else {
- if (castType.isPresent()) {
- result.add(Operand.of(flag,
TypeCoercionUtils.castIfNotSameType(expr, castType.get())));
- } else {
- result.add(Operand.of(flag, expr));
- }
- }
- }
-
- private static boolean hasConstantOperand(Expression expr, boolean
isAddOrSub) {
- if (expr.isConstant()) {
- return true;
- }
-
- Predicate<Expression> checkArithmetic = isAddOrSub
- ? TypeUtils::isAddOrSubtract : TypeUtils::isMultiplyOrDivide;
- BinaryArithmetic arithmetic = null;
- if (checkArithmetic.test(expr)) {
- arithmetic = (BinaryArithmetic) expr;
- } else if (expr instanceof Cast) {
- Cast cast = (Cast) expr;
- if (checkArithmetic.test(cast.child())) {
- arithmetic = (BinaryArithmetic) cast.child();
- }
- }
- if (arithmetic != null) {
- return hasConstantOperand(arithmetic.left(), isAddOrSub)
- || hasConstantOperand(arithmetic.right(), isAddOrSub);
- } else {
- return false;
+ // Keep non-arithmetic expressions atomic. In particular, moving a
cast to the
+ // operands can change rounding, overflow, error and null behavior.
+ result.add(Operand.of(flag, expr));
}
}
@@ -241,4 +204,3 @@ public class SimplifyArithmeticRule implements
ExpressionPatternRuleFactory {
}
}
}
-
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/SimplifyArithmeticRuleTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/SimplifyArithmeticRuleTest.java
index df14840f628..cb09906214b 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/SimplifyArithmeticRuleTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/SimplifyArithmeticRuleTest.java
@@ -95,6 +95,19 @@ class SimplifyArithmeticRuleTest extends
ExpressionRewriteTestHelper {
assertRewriteAfterSimplify("-IA / 2.0 * ((-IB - 1) - (3 + (IC + 4)))",
"(((0 - IA) / 2.0) * (((0 - IB) - 1) - (3 + (IC + 4))))");
}
+ @Test
+ void testPreserveCastAroundArithmetic() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(
+ bottomUp(SimplifyArithmeticRule.INSTANCE)
+ ));
+
+ String castExpression = "cast(LA - 9223372036854775800 as double) +
cast(0 as double)";
+ assertRewriteAfterSimplify(castExpression, castExpression);
+
+ String tryCastExpression = "try_cast(LA - 9223372036854775800 as
double) + cast(0 as double)";
+ assertRewriteAfterSimplify(tryCastExpression, tryCastExpression);
+ }
+
@Test
void testSimplifyArithmeticComparison() {
executor = new ExpressionRuleExecutor(ImmutableList.of(
diff --git
a/regression-test/suites/external_table_p0/dialect_compatible/sql/presto/scalar/timestamp/TestOperators.sql
b/regression-test/suites/external_table_p0/dialect_compatible/sql/presto/scalar/timestamp/TestOperators.sql
index 5a3d008a0e4..c8510432786 100644
---
a/regression-test/suites/external_table_p0/dialect_compatible/sql/presto/scalar/timestamp/TestOperators.sql
+++
b/regression-test/suites/external_table_p0/dialect_compatible/sql/presto/scalar/timestamp/TestOperators.sql
@@ -222,6 +222,7 @@ SELECT TIMESTAMP '2020-05-01 12:34:56.123456789' BETWEEN
TIMESTAMP '2020-05-01 1
SELECT TIMESTAMP '2020-05-01 12:34:56.1234567890' BETWEEN TIMESTAMP
'2020-05-01 12:34:56.1234567889' and TIMESTAMP '2020-05-01 12:34:56.1234567891';
SELECT TIMESTAMP '2020-05-01 12:34:56.12345678901' BETWEEN TIMESTAMP
'2020-05-01 12:34:56.1234567890' and TIMESTAMP '2020-05-01
12:34:56.12345678902';
SELECT TIMESTAMP '2020-05-01 12:34:56.123456789012' BETWEEN TIMESTAMP
'2020-05-01 12:34:56.123456789011' and TIMESTAMP '2020-05-01
12:34:56.123456789013';
+set debug_skip_fold_constant=true;
-- SELECT TIMESTAMP '2020-05-01 12:34:56' + INTERVAL '1.123' SECOND; # differ:
doris : None, presto : 2020-05-01 12:34:57.123
-- SELECT TIMESTAMP '2020-05-01 12:34:56.1' + INTERVAL '1.123' SECOND; #
differ: doris : None, presto : 2020-05-01 12:34:57.223
-- SELECT TIMESTAMP '2020-05-01 12:34:56.12' + INTERVAL '1.123' SECOND; #
differ: doris : None, presto : 2020-05-01 12:34:57.243
@@ -389,7 +390,6 @@ SELECT TIMESTAMP '2020-05-01 12:34:56.123456789012' BETWEEN
TIMESTAMP '2020-05-0
-- SELECT TIMESTAMP '2020-05-01 12:34:55.1111111111' - TIMESTAMP '2020-05-01
12:34:56.9999999999'; # differ: doris : -2, presto : -0 00:00:01.889
-- SELECT TIMESTAMP '2020-05-01 12:34:55.11111111111' - TIMESTAMP '2020-05-01
12:34:56.99999999999'; # differ: doris : -2, presto : -0 00:00:01.889
-- SELECT TIMESTAMP '2020-05-01 12:34:55.111111111111' - TIMESTAMP '2020-05-01
12:34:56.999999999999'; # differ: doris : -2, presto : -0 00:00:01.889
-set debug_skip_fold_constant=true;
SELECT TIMESTAMP '2020-05-01 12:34:56' = TIMESTAMP '2020-05-01 12:34:56';
SELECT TIMESTAMP '2020-05-01 12:34:56.1' = TIMESTAMP '2020-05-01 12:34:56.1';
SELECT TIMESTAMP '2020-05-01 12:34:56.12' = TIMESTAMP '2020-05-01 12:34:56.12';
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
