This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 668ef05db4b1 [SPARK-60868][SQL] Fix wrong results by ConstantFolding 
for size of an array with a failing LEGACY cast
668ef05db4b1 is described below

commit 668ef05db4b178cfb939d4362b87120429d5b386
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 09:49:14 2026 -0800

    [SPARK-60868][SQL] Fix wrong results by ConstantFolding for size of an 
array with a failing LEGACY cast
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix a correctness issue where `ConstantFolding` returns 
wrong results for `size(array(...))` over a failing LEGACY cast when 
`spark.sql.ansi.enabled` is `false`.
    
    `ConstantFolding.hasNoSideEffect` treats every non-ANSI `Cast` as free of 
side effects under a non-ANSI session, so `size(array(...))` over such a cast 
is folded to the number of elements without evaluating the cast. However, some 
LEGACY casts still fail at runtime, for example:
    - `CAST(INTERVAL '1000' SECOND AS TINYINT)` fails with `CAST_OVERFLOW`.
    - `CAST('abc' AS INTERVAL DAY TO SECOND)` fails with 
`INVALID_INTERVAL_FORMAT`.
    - `CAST(12345 AS VARCHAR(2))` fails with `EXCEED_LIMIT_LENGTH` when 
`spark.sql.preserveCharVarcharTypeInfo` or 
`spark.sql.charVarchar.standardSemantics.enabled` is `true`.
    
    When the error is caught, for example by `try_add` of a date and a number, 
the folded size gives a wrong value instead of `NULL`. Otherwise, the query 
returns the size instead of failing.
    
    This PR adds `Cast.isNoThrowLegacyCast`, which accepts only the LEGACY 
casts among numeric, boolean and string types, without a CHAR/VARCHAR length 
constraint on the target. These casts wrap around or saturate on overflow, or 
return null, instead of failing. `ConstantFolding` now skips only such LEGACY 
casts and TRY casts, which return null instead of failing.
    
    The session check stays, so this PR does not fold anything new in ANSI 
sessions. It only stops folding the LEGACY casts that are not accepted by 
`Cast.isNoThrowLegacyCast` in non-ANSI sessions.
    
    This PR also adds a migration guide entry.
    
    ### Why are the changes needed?
    
    To fix a correctness issue. This bug was introduced at Apache Spark 3.1.3 
and 3.2.1 by SPARK-37392 (#34823), which started to skip every cast in 
`ConstantFolding.hasNoSideEffect` under a non-ANSI session.
    - https://github.com/apache/spark/releases/tag/v3.1.3 (2022-02-06)
    - https://github.com/apache/spark/releases/tag/v3.2.1 (2022-01-20)
    
    ```sql
    SET spark.sql.ansi.enabled=false;
    CREATE TABLE t(i INTERVAL SECOND) USING parquet;
    INSERT INTO t VALUES (INTERVAL '1000' SECOND);
    
    -- `try_add` returns NULL when an argument fails.
    SELECT try_add(DATE'2020-01-01', CAST(i AS TINYINT)) FROM t;
    -- NULL
    
    -- Issue 1: It should return NULL, but returns 2020-01-02.
    SELECT try_add(DATE'2020-01-01', size(array(CAST(i AS TINYINT)))) FROM t;
    
    -- Issue 2: It should fail with CAST_OVERFLOW, but returns 1.
    SELECT size(array(CAST(i AS TINYINT))) FROM t;
    ```
    
    The case of SPARK-37392, `size(array(CAST(int_col AS STRING), ...))`, is 
still folded.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. When `spark.sql.ansi.enabled` is `false`, `size()` and `cardinality()` 
of an array built from a failing LEGACY cast now fail with the cast error 
instead of returning the array size, or returns `NULL` when the error is 
caught, for example by `try_add`. The result is unchanged when the cast does 
not fail. The migration guide is updated.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60083 from dongjoon-hyun/SPARK-60868.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                           |  1 +
 .../apache/spark/sql/catalyst/expressions/Cast.scala  | 15 +++++++++++++++
 .../spark/sql/catalyst/optimizer/expressions.scala    | 11 ++++++++---
 .../sql/catalyst/optimizer/ConstantFoldingSuite.scala | 19 +++++++++++++++++++
 4 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 9288c532b66e..148cd24c08db 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -31,6 +31,7 @@ license: |
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `DECIMAL` type from directory names, it converts each value to 
the precision and scale of that type, rounding half up as `CAST` does. 
Previously, the vectorized Parquet and ORC readers applied the declared scale 
to the digits in the directory name without an error, for example `d=1.5` read 
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results 
may need to be recomputed. A value that [...]
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out 
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of 
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped 
around without an error, for example `b=300` read as `TINYINT` returned `44`. 
Data derived from affected query results may need to be recomputed. To get 
`NULL` for such a value, set `spark. [...]
 - Since Spark 4.4, `size` and `cardinality` of a map built by `map(...)` 
return the number of entries of the built map. Previously, the optimizer could 
return the number of key-value pairs instead, for example `size(map(1, 'a', 1, 
'b'))` returned `2` instead of `1` when `spark.sql.mapKeyDedupPolicy` is 
`LAST_WIN`. Data derived from affected query results may need to be recomputed. 
Under the default `EXCEPTION` policy, such a query now fails with 
`DUPLICATED_MAP_KEY`, and a NULL key fails [...]
+- Since Spark 4.4, when `spark.sql.ansi.enabled` is `false`, `size` and 
`cardinality` of an array built by `array(...)` evaluate the LEGACY casts in it 
that can fail, such as a cast from an interval to `TINYINT`, from a string to 
an interval, or, when `spark.sql.preserveCharVarcharTypeInfo` or 
`spark.sql.charVarchar.standardSemantics.enabled` is `true`, to `VARCHAR(n)`. 
Previously, the optimizer could return the number of elements without 
evaluating such a cast. For example, for an `INTE [...]
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory 
names, a value that cannot be cast to that type fails with 
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously, 
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it 
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such 
as `2020-01-01 25:00:00`, became the start of that da [...]
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index c41bcbf76ad3..ed8fa99074c4 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -594,6 +594,21 @@ object Cast extends QueryErrorsBase {
     case _ => false
   }
 
+  /**
+   * Returns true if a LEGACY cast from `from` type to `to` type never fails 
at runtime. Among
+   * numeric, boolean and string types, except to CHAR/VARCHAR, it wraps 
around or saturates on
+   * overflow, or returns null, instead of failing. Returns false for other 
LEGACY casts, some of
+   * which can fail, for example from an interval to a number, from a string 
to an interval, and
+   * to CHAR/VARCHAR. The patterns matched reflect the current implementation 
in the Cast node and
+   * must stay in sync with it.
+   */
+  def isNoThrowLegacyCast(from: DataType, to: DataType): Boolean = (from, to) 
match {
+    case (_: NumericType | BooleanType | _: StringType, _: NumericType | 
BooleanType) => true
+    case (_: NumericType | BooleanType | _: StringType, s: StringType) =>
+      StringHelper.isPlainString(s)
+    case _ => false
+  }
+
   def resolvableNullability(from: Boolean, to: Boolean): Boolean = !from || to
 
   /**
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 78ccafd6f027..842c8e6fed07 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -56,9 +56,14 @@ object ConstantFolding extends Rule[LogicalPlan] {
     case _: Attribute => true
     case _: Literal => true
     // A cast keeps the eval mode it was analyzed with (for example, in a 
view), so check it as
-    // well. Keep the session check: some LEGACY casts can still fail, such as 
an interval
-    // overflow, and this should not start folding them in ANSI sessions.
-    case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI => 
hasNoSideEffect(c.child)
+    // well. A TRY cast returns null instead of failing, and so do some LEGACY 
casts. Keep the
+    // session check so that this does not start folding casts in ANSI 
sessions.
+    case c: Cast if !conf.ansiEnabled => c.evalMode match {
+      case EvalMode.TRY => hasNoSideEffect(c.child)
+      case EvalMode.LEGACY if Cast.isNoThrowLegacyCast(c.child.dataType, 
c.dataType) =>
+        hasNoSideEffect(c.child)
+      case _ => false
+    }
     case _: NoThrow if e.deterministic => e.children.forall(hasNoSideEffect)
     case _ => false
   }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index 997996d28965..f9f93eb1165e 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -342,6 +342,25 @@ class ConstantFoldingSuite extends PlanTest {
     }
   }
 
+  test("SPARK-60868: Do not fold the size of CreateArray over a LEGACY cast 
that can fail") {
+    val t = LocalRelation($"i".int, $"d".dayTimeInterval())
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+      Seq(
+        // These cannot fail: a LEGACY cast as in SPARK-37392, and a TRY cast 
that returns null.
+        Cast($"i", StringType) -> true,
+        Cast($"d", ByteType, evalMode = EvalMode.TRY) -> true,
+        // These fail with CAST_OVERFLOW, EXCEED_LIMIT_LENGTH, and 
CAST_OVERFLOW in the child.
+        Cast($"d", ByteType) -> false,
+        Cast($"i", VarcharType(2)) -> false,
+        Cast(Cast($"d", ByteType), StringType) -> false).foreach { case (cast, 
foldable) =>
+        assert(cast.evalMode != EvalMode.ANSI)
+        val query = t.select(Size(CreateArray(Seq(cast))).as("s")).analyze
+        val expected = if (foldable) t.select(Literal(1).as("s")).analyze else 
query
+        comparePlans(Optimize.execute(query), expected)
+      }
+    }
+  }
+
   test("SPARK-60849: Constant folding test CreateMap") {
     // Building a map deduplicates its keys, so its size can be less than the 
number of pairs.
     withSQLConf(SQLConf.MAP_KEY_DEDUP_POLICY.key -> 
SQLConf.MapKeyDedupPolicy.LAST_WIN.toString) {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to