This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2c03f1681fb4 [SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast
errors when the session has ANSI mode off
2c03f1681fb4 is described below
commit 2c03f1681fb44bfac5530aeb8f8331582fe90c28
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 01:39:31 2026 -0800
[SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast errors when the
session has ANSI mode off
### What changes were proposed in this pull request?
`ConstantFolding` folds `size(array(...))` and `size(map(...))` into the
number of elements when every element is free of side effects.
`hasNoSideEffect` treated a `Cast` as free of side effects whenever the session
had ANSI mode off, without looking at the cast's own eval mode. A cast keeps
the eval mode it was analyzed with, so an ANSI cast analyzed with ANSI on (in a
view, a SQL function, or an eagerly analyzed DataFrame) was folded away when
the optimizer ran under a non-ANSI sess [...]
This PR also checks the cast's own eval mode, so an ANSI cast is never
treated as free of side effects:
```scala
case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
```
The session check stays. Some LEGACY casts can still fail, such as
`CAST(INTERVAL '1000' SECOND AS TINYINT)` (`CAST_OVERFLOW`). Checking only the
cast's eval mode would start folding them in ANSI sessions, where they are not
folded today, and drop those errors, for example for a view created with ANSI
off and queried with ANSI on.
### Why are the changes needed?
The last query below should fail with `CAST_INVALID_INPUT` like the one on
`v2`, but returns `1`.
```sql
CREATE TABLE src(a STRING) USING parquet;
INSERT INTO src VALUES ('abc');
SET spark.sql.ansi.enabled=true;
CREATE VIEW v1 AS SELECT size(array(CAST(a AS INT))) AS d FROM src;
CREATE VIEW v2 AS SELECT CAST(a AS INT) AS d FROM src;
SET spark.sql.ansi.enabled=false;
SELECT * FROM v2; -- CAST_INVALID_INPUT
SELECT * FROM v1; -- It should fail with CAST_INVALID_INPUT, but returns 1
```
A SQL function created with ANSI on and a DataFrame analyzed with ANSI on
behave the same when they run with ANSI off.
This was found in the review of #59965, which analyzes the generation
expression of a DSv2 generated column with ANSI on. With that PR, a column `d
INT GENERATED ALWAYS AS (size(array(CAST(a AS INT))))` still accepts `INSERT
INTO t (a) VALUES ('abc')` from a non-ANSI writer and stores `d = 1`, while an
ANSI writer gets `CAST_INVALID_INPUT`.
The session-only check was added by SPARK-37392 (#34823) and has been in
Spark since [v3.3.0](https://github.com/apache/spark/tree/v3.3.0) (2022-06-09).
### Does this PR introduce _any_ user-facing change?
Yes. Under a non-ANSI session, `size(array(...))` and `size(map(...))` over
an ANSI cast now raise the cast's error instead of returning the number of
elements. Nothing changes under an ANSI session, and LEGACY and TRY casts are
folded as before.
### How was this patch tested?
- Added `SPARK-60820: Do not fold the size of CreateArray over an ANSI cast
in a non-ANSI session` to `ConstantFoldingSuite`. It checks every eval mode of
a cast under ANSI on and off, and only LEGACY and TRY casts under a non-ANSI
session are folded. Without this change, the ANSI cast under a non-ANSI session
is folded to `1` and the test fails.
- Ran all suites in `org.apache.spark.sql.catalyst.optimizer`,
`SQLQueryTestSuite` for `array.sql`, `cast.sql`, `try_cast.sql`, `explain*.sql`
and the `nonansi` variants, `GeneratorFunctionSuite`, `SimpleSQLViewSuite` and
`DataFrameFunctionsSuite`. All pass and no golden files change.
- Ran the view, SQL function and DataFrame cases above before and after
this change. With this change, they fail with `CAST_INVALID_INPUT` under a
non-ANSI session. A view created with ANSI off that casts an overflowing
interval to `TINYINT` still fails with `CAST_OVERFLOW` under an ANSI session,
as before.
- Applied this change on top of #59965 and ran the generated column case
above. Both writers now get `CAST_INVALID_INPUT`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60038 from dongjoon-hyun/SPARK-60820.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/optimizer/expressions.scala | 5 ++++-
.../sql/catalyst/optimizer/ConstantFoldingSuite.scala | 18 ++++++++++++++++++
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 483c48e0cfee..9e8c5fba4135 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -55,7 +55,10 @@ object ConstantFolding extends Rule[LogicalPlan] {
private def hasNoSideEffect(e: Expression): Boolean = e match {
case _: Attribute => true
case _: Literal => true
- case c: Cast if !conf.ansiEnabled => hasNoSideEffect(c.child)
+ // A cast keeps the eval mode it was analyzed with (for example, in a
view), so check it as
+ // well. Keep the session check: some LEGACY casts can still fail, such as
an interval
+ // overflow, and this should not start folding them in ANSI sessions.
+ case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
case _: NoThrow if e.deterministic => e.children.forall(hasNoSideEffect)
case _ => false
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index eb9c43f204a8..e160accec9ac 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -324,6 +324,24 @@ class ConstantFoldingSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ test("SPARK-60820: Do not fold the size of CreateArray over an ANSI cast in
a non-ANSI session") {
+ val t = LocalRelation($"s".string)
+ Seq(false, true).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ EvalMode.values.foreach { evalMode =>
+ val cast = Cast($"s", IntegerType, evalMode = evalMode)
+ val query = t.select(Size(CreateArray(Seq(cast))).as("d")).analyze
+ val expected = if (!ansiEnabled && evalMode != EvalMode.ANSI) {
+ t.select(Literal(1).as("d")).analyze
+ } else {
+ query
+ }
+ comparePlans(Optimize.execute(query), expected)
+ }
+ }
+ }
+ }
+
test("SPARK-37907: InvokeLike support ConstantFolding") {
val originalQuery =
testRelation
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]