This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new deab27835099 [SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast
errors when the session has ANSI mode off
deab27835099 is described below
commit deab27835099c8cc975fbe308dc4a7273a31ea28
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 01:39:31 2026 -0800
[SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast errors when the
session has ANSI mode off
### What changes were proposed in this pull request?
`ConstantFolding` folds `size(array(...))` and `size(map(...))` into the
number of elements when every element is free of side effects.
`hasNoSideEffect` treated a `Cast` as free of side effects whenever the session
had ANSI mode off, without looking at the cast's own eval mode. A cast keeps
the eval mode it was analyzed with, so an ANSI cast analyzed with ANSI on (in a
view, a SQL function, or an eagerly analyzed DataFrame) was folded away when
the optimizer ran under a non-ANSI sess [...]
This PR also checks the cast's own eval mode, so an ANSI cast is never
treated as free of side effects:
```scala
case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
```
The session check stays. Some LEGACY casts can still fail, such as
`CAST(INTERVAL '1000' SECOND AS TINYINT)` (`CAST_OVERFLOW`). Checking only the
cast's eval mode would start folding them in ANSI sessions, where they are not
folded today, and drop those errors, for example for a view created with ANSI
off and queried with ANSI on.
### Why are the changes needed?
The last query below should fail with `CAST_INVALID_INPUT` like the one on
`v2`, but returns `1`.
```sql
CREATE TABLE src(a STRING) USING parquet;
INSERT INTO src VALUES ('abc');
SET spark.sql.ansi.enabled=true;
CREATE VIEW v1 AS SELECT size(array(CAST(a AS INT))) AS d FROM src;
CREATE VIEW v2 AS SELECT CAST(a AS INT) AS d FROM src;
SET spark.sql.ansi.enabled=false;
SELECT * FROM v2; -- CAST_INVALID_INPUT
SELECT * FROM v1; -- It should fail with CAST_INVALID_INPUT, but returns 1
```
A SQL function created with ANSI on and a DataFrame analyzed with ANSI on
behave the same when they run with ANSI off.
This was found in the review of #59965, which analyzes the generation
expression of a DSv2 generated column with ANSI on. With that PR, a column `d
INT GENERATED ALWAYS AS (size(array(CAST(a AS INT))))` still accepts `INSERT
INTO t (a) VALUES ('abc')` from a non-ANSI writer and stores `d = 1`, while an
ANSI writer gets `CAST_INVALID_INPUT`.
The session-only check was added by SPARK-37392 (#34823) and has been in
Spark since [v3.3.0](https://github.com/apache/spark/tree/v3.3.0) (2022-06-09).
### Does this PR introduce _any_ user-facing change?
Yes. Under a non-ANSI session, `size(array(...))` and `size(map(...))` over
an ANSI cast now raise the cast's error instead of returning the number of
elements. Nothing changes under an ANSI session, and LEGACY and TRY casts are
folded as before.
### How was this patch tested?
- Added `SPARK-60820: Do not fold the size of CreateArray over an ANSI cast
in a non-ANSI session` to `ConstantFoldingSuite`. It checks every eval mode of
a cast under ANSI on and off, and only LEGACY and TRY casts under a non-ANSI
session are folded. Without this change, the ANSI cast under a non-ANSI session
is folded to `1` and the test fails.
- Ran all suites in `org.apache.spark.sql.catalyst.optimizer`,
`SQLQueryTestSuite` for `array.sql`, `cast.sql`, `try_cast.sql`, `explain*.sql`
and the `nonansi` variants, `GeneratorFunctionSuite`, `SimpleSQLViewSuite` and
`DataFrameFunctionsSuite`. All pass and no golden files change.
- Ran the view, SQL function and DataFrame cases above before and after
this change. With this change, they fail with `CAST_INVALID_INPUT` under a
non-ANSI session. A view created with ANSI off that casts an overflowing
interval to `TINYINT` still fails with `CAST_OVERFLOW` under an ANSI session,
as before.
- Applied this change on top of #59965 and ran the generated column case
above. Both writers now get `CAST_INVALID_INPUT`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60038 from dongjoon-hyun/SPARK-60820.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 2c03f1681fb44bfac5530aeb8f8331582fe90c28)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 7f99034f1e09beb19ef0279583fe9c8e144f0ce8)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f6998647e13cf0c8f4b2f75cfc64423d49b41192)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/optimizer/expressions.scala | 5 ++++-
.../sql/catalyst/optimizer/ConstantFoldingSuite.scala | 18 ++++++++++++++++++
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 25b475b7ba1c..6c0c21836e1f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -55,7 +55,10 @@ object ConstantFolding extends Rule[LogicalPlan] {
private def hasNoSideEffect(e: Expression): Boolean = e match {
case _: Attribute => true
case _: Literal => true
- case c: Cast if !conf.ansiEnabled => hasNoSideEffect(c.child)
+ // A cast keeps the eval mode it was analyzed with (for example, in a
view), so check it as
+ // well. Keep the session check: some LEGACY casts can still fail, such as
an interval
+ // overflow, and this should not start folding them in ANSI sessions.
+ case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
case _: NoThrow if e.deterministic => e.children.forall(hasNoSideEffect)
case _ => false
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index 0489308a1ab2..972f0ddf1586 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -307,6 +307,24 @@ class ConstantFoldingSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ test("SPARK-60820: Do not fold the size of CreateArray over an ANSI cast in
a non-ANSI session") {
+ val t = LocalRelation($"s".string)
+ Seq(false, true).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ EvalMode.values.foreach { evalMode =>
+ val cast = Cast($"s", IntegerType, evalMode = evalMode)
+ val query = t.select(Size(CreateArray(Seq(cast))).as("d")).analyze
+ val expected = if (!ansiEnabled && evalMode != EvalMode.ANSI) {
+ t.select(Literal(1).as("d")).analyze
+ } else {
+ query
+ }
+ comparePlans(Optimize.execute(query), expected)
+ }
+ }
+ }
+ }
+
test("SPARK-37907: InvokeLike support ConstantFolding") {
val originalQuery =
testRelation
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]