This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9077a47c876a [SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast 
errors when the session has ANSI mode off
9077a47c876a is described below

commit 9077a47c876ae1029b919c779d85f44dbb3d3fb6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 01:39:31 2026 -0800

    [SPARK-60820][SQL] Fix ConstantFolding dropping ANSI cast errors when the 
session has ANSI mode off
    
    ### What changes were proposed in this pull request?
    
    `ConstantFolding` folds `size(array(...))` and `size(map(...))` into the 
number of elements when every element is free of side effects. 
`hasNoSideEffect` treated a `Cast` as free of side effects whenever the session 
had ANSI mode off, without looking at the cast's own eval mode. A cast keeps 
the eval mode it was analyzed with, so an ANSI cast analyzed with ANSI on (in a 
view, a SQL function, or an eagerly analyzed DataFrame) was folded away when 
the optimizer ran under a non-ANSI sess [...]
    
    This PR also checks the cast's own eval mode, so an ANSI cast is never 
treated as free of side effects:
    
    ```scala
    case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI => 
hasNoSideEffect(c.child)
    ```
    
    The session check stays. Some LEGACY casts can still fail, such as 
`CAST(INTERVAL '1000' SECOND AS TINYINT)` (`CAST_OVERFLOW`). Checking only the 
cast's eval mode would start folding them in ANSI sessions, where they are not 
folded today, and drop those errors, for example for a view created with ANSI 
off and queried with ANSI on.
    
    ### Why are the changes needed?
    
    The last query below should fail with `CAST_INVALID_INPUT` like the one on 
`v2`, but returns `1`.
    
    ```sql
    CREATE TABLE src(a STRING) USING parquet;
    INSERT INTO src VALUES ('abc');
    
    SET spark.sql.ansi.enabled=true;
    CREATE VIEW v1 AS SELECT size(array(CAST(a AS INT))) AS d FROM src;
    CREATE VIEW v2 AS SELECT CAST(a AS INT) AS d FROM src;
    
    SET spark.sql.ansi.enabled=false;
    SELECT * FROM v2;  -- CAST_INVALID_INPUT
    SELECT * FROM v1;  -- It should fail with CAST_INVALID_INPUT, but returns 1
    ```
    
    A SQL function created with ANSI on and a DataFrame analyzed with ANSI on 
behave the same when they run with ANSI off.
    
    This was found in the review of #59965, which analyzes the generation 
expression of a DSv2 generated column with ANSI on. With that PR, a column `d 
INT GENERATED ALWAYS AS (size(array(CAST(a AS INT))))` still accepts `INSERT 
INTO t (a) VALUES ('abc')` from a non-ANSI writer and stores `d = 1`, while an 
ANSI writer gets `CAST_INVALID_INPUT`.
    
    The session-only check was added by SPARK-37392 (#34823) and has been in 
Spark since [v3.3.0](https://github.com/apache/spark/tree/v3.3.0) (2022-06-09).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Under a non-ANSI session, `size(array(...))` and `size(map(...))` over 
an ANSI cast now raise the cast's error instead of returning the number of 
elements. Nothing changes under an ANSI session, and LEGACY and TRY casts are 
folded as before.
    
    ### How was this patch tested?
    
    - Added `SPARK-60820: Do not fold the size of CreateArray over an ANSI cast 
in a non-ANSI session` to `ConstantFoldingSuite`. It checks every eval mode of 
a cast under ANSI on and off, and only LEGACY and TRY casts under a non-ANSI 
session are folded. Without this change, the ANSI cast under a non-ANSI session 
is folded to `1` and the test fails.
    - Ran all suites in `org.apache.spark.sql.catalyst.optimizer`, 
`SQLQueryTestSuite` for `array.sql`, `cast.sql`, `try_cast.sql`, `explain*.sql` 
and the `nonansi` variants, `GeneratorFunctionSuite`, `SimpleSQLViewSuite` and 
`DataFrameFunctionsSuite`. All pass and no golden files change.
    - Ran the view, SQL function and DataFrame cases above before and after 
this change. With this change, they fail with `CAST_INVALID_INPUT` under a 
non-ANSI session. A view created with ANSI off that casts an overflowing 
interval to `TINYINT` still fails with `CAST_OVERFLOW` under an ANSI session, 
as before.
    - Applied this change on top of #59965 and ran the generated column case 
above. Both writers now get `CAST_INVALID_INPUT`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60038 from dongjoon-hyun/SPARK-60820.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 2c03f1681fb44bfac5530aeb8f8331582fe90c28)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/optimizer/expressions.scala     |  5 ++++-
 .../sql/catalyst/optimizer/ConstantFoldingSuite.scala  | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 483c48e0cfee..9e8c5fba4135 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -55,7 +55,10 @@ object ConstantFolding extends Rule[LogicalPlan] {
   private def hasNoSideEffect(e: Expression): Boolean = e match {
     case _: Attribute => true
     case _: Literal => true
-    case c: Cast if !conf.ansiEnabled => hasNoSideEffect(c.child)
+    // A cast keeps the eval mode it was analyzed with (for example, in a 
view), so check it as
+    // well. Keep the session check: some LEGACY casts can still fail, such as 
an interval
+    // overflow, and this should not start folding them in ANSI sessions.
+    case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI => 
hasNoSideEffect(c.child)
     case _: NoThrow if e.deterministic => e.children.forall(hasNoSideEffect)
     case _ => false
   }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index eb9c43f204a8..e160accec9ac 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -324,6 +324,24 @@ class ConstantFoldingSuite extends PlanTest {
     comparePlans(optimized, correctAnswer)
   }
 
+  test("SPARK-60820: Do not fold the size of CreateArray over an ANSI cast in 
a non-ANSI session") {
+    val t = LocalRelation($"s".string)
+    Seq(false, true).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        EvalMode.values.foreach { evalMode =>
+          val cast = Cast($"s", IntegerType, evalMode = evalMode)
+          val query = t.select(Size(CreateArray(Seq(cast))).as("d")).analyze
+          val expected = if (!ansiEnabled && evalMode != EvalMode.ANSI) {
+            t.select(Literal(1).as("d")).analyze
+          } else {
+            query
+          }
+          comparePlans(Optimize.execute(query), expected)
+        }
+      }
+    }
+  }
+
   test("SPARK-37907: InvokeLike support ConstantFolding") {
     val originalQuery =
       testRelation


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to