Yu Xu created SPARK-56940:
-----------------------------
Summary: Extend OptimizeRand optimizer to support arithmetic
expressions
Key: SPARK-56940
URL: https://issues.apache.org/jira/browse/SPARK-56940
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.1
Reporter: Yu Xu
Fix For: 4.1.2
The Apache Spark SQL optimizer rule OptimizeRand only handled simple
comparisons with the rand() function, such as rand() > 0.5 or rand() < 1. It
did not optimize more complex expressions that
combine rand() with arithmetic operations, such as:
- rand() * 3 < 3 (scalar multiplication)
- rand() + 1 < 2 (addition)
- rand() - 1 < -1 (subtraction)
- rand() / 2 < 1 (division)
- Nested expressions: 2 * rand() + 1 < 3
These queries can still be statically evaluated because rand() always produces
values in the range [0, 1), making the outcome of many such comparisons
deterministic at compile time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
