This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 668ef05db4b1 [SPARK-60868][SQL] Fix wrong results by ConstantFolding
for size of an array with a failing LEGACY cast
668ef05db4b1 is described below
commit 668ef05db4b178cfb939d4362b87120429d5b386
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 09:49:14 2026 -0800
[SPARK-60868][SQL] Fix wrong results by ConstantFolding for size of an
array with a failing LEGACY cast
### What changes were proposed in this pull request?
This PR aims to fix a correctness issue where `ConstantFolding` returns
wrong results for `size(array(...))` over a failing LEGACY cast when
`spark.sql.ansi.enabled` is `false`.
`ConstantFolding.hasNoSideEffect` treats every non-ANSI `Cast` as free of
side effects under a non-ANSI session, so `size(array(...))` over such a cast
is folded to the number of elements without evaluating the cast. However, some
LEGACY casts still fail at runtime, for example:
- `CAST(INTERVAL '1000' SECOND AS TINYINT)` fails with `CAST_OVERFLOW`.
- `CAST('abc' AS INTERVAL DAY TO SECOND)` fails with
`INVALID_INTERVAL_FORMAT`.
- `CAST(12345 AS VARCHAR(2))` fails with `EXCEED_LIMIT_LENGTH` when
`spark.sql.preserveCharVarcharTypeInfo` or
`spark.sql.charVarchar.standardSemantics.enabled` is `true`.
When the error is caught, for example by `try_add` of a date and a number,
the folded size gives a wrong value instead of `NULL`. Otherwise, the query
returns the size instead of failing.
This PR adds `Cast.isNoThrowLegacyCast`, which accepts only the LEGACY
casts among numeric, boolean and string types, without a CHAR/VARCHAR length
constraint on the target. These casts wrap around or saturate on overflow, or
return null, instead of failing. `ConstantFolding` now skips only such LEGACY
casts and TRY casts, which return null instead of failing.
The session check stays, so this PR does not fold anything new in ANSI
sessions. It only stops folding the LEGACY casts that are not accepted by
`Cast.isNoThrowLegacyCast` in non-ANSI sessions.
This PR also adds a migration guide entry.
### Why are the changes needed?
To fix a correctness issue. This bug was introduced at Apache Spark 3.1.3
and 3.2.1 by SPARK-37392 (#34823), which started to skip every cast in
`ConstantFolding.hasNoSideEffect` under a non-ANSI session.
- https://github.com/apache/spark/releases/tag/v3.1.3 (2022-02-06)
- https://github.com/apache/spark/releases/tag/v3.2.1 (2022-01-20)
```sql
SET spark.sql.ansi.enabled=false;
CREATE TABLE t(i INTERVAL SECOND) USING parquet;
INSERT INTO t VALUES (INTERVAL '1000' SECOND);
-- `try_add` returns NULL when an argument fails.
SELECT try_add(DATE'2020-01-01', CAST(i AS TINYINT)) FROM t;
-- NULL
-- Issue 1: It should return NULL, but returns 2020-01-02.
SELECT try_add(DATE'2020-01-01', size(array(CAST(i AS TINYINT)))) FROM t;
-- Issue 2: It should fail with CAST_OVERFLOW, but returns 1.
SELECT size(array(CAST(i AS TINYINT))) FROM t;
```
The case of SPARK-37392, `size(array(CAST(int_col AS STRING), ...))`, is
still folded.
### Does this PR introduce _any_ user-facing change?
Yes. When `spark.sql.ansi.enabled` is `false`, `size()` and `cardinality()`
of an array built from a failing LEGACY cast now fail with the cast error
instead of returning the array size, or returns `NULL` when the error is
caught, for example by `try_add`. The result is unchanged when the cast does
not fail. The migration guide is updated.
### How was this patch tested?
Pass the CIs with the newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60083 from dongjoon-hyun/SPARK-60868.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../apache/spark/sql/catalyst/expressions/Cast.scala | 15 +++++++++++++++
.../spark/sql/catalyst/optimizer/expressions.scala | 11 ++++++++---
.../sql/catalyst/optimizer/ConstantFoldingSuite.scala | 19 +++++++++++++++++++
4 files changed, 43 insertions(+), 3 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 9288c532b66e..148cd24c08db 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -31,6 +31,7 @@ license: |
- Since Spark 4.4, when a file source reads a partition column with a
user-specified `DECIMAL` type from directory names, it converts each value to
the precision and scale of that type, rounding half up as `CAST` does.
Previously, the vectorized Parquet and ORC readers applied the declared scale
to the digits in the directory name without an error, for example `d=1.5` read
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results
may need to be recomputed. A value that [...]
- Since Spark 4.4, when a file source reads a partition column with a
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped
around without an error, for example `b=300` read as `TINYINT` returned `44`.
Data derived from affected query results may need to be recomputed. To get
`NULL` for such a value, set `spark. [...]
- Since Spark 4.4, `size` and `cardinality` of a map built by `map(...)`
return the number of entries of the built map. Previously, the optimizer could
return the number of key-value pairs instead, for example `size(map(1, 'a', 1,
'b'))` returned `2` instead of `1` when `spark.sql.mapKeyDedupPolicy` is
`LAST_WIN`. Data derived from affected query results may need to be recomputed.
Under the default `EXCEPTION` policy, such a query now fails with
`DUPLICATED_MAP_KEY`, and a NULL key fails [...]
+- Since Spark 4.4, when `spark.sql.ansi.enabled` is `false`, `size` and
`cardinality` of an array built by `array(...)` evaluate the LEGACY casts in it
that can fail, such as a cast from an interval to `TINYINT`, from a string to
an interval, or, when `spark.sql.preserveCharVarcharTypeInfo` or
`spark.sql.charVarchar.standardSemantics.enabled` is `true`, to `VARCHAR(n)`.
Previously, the optimizer could return the number of elements without
evaluating such a cast. For example, for an `INTE [...]
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, when a file source reads a partition column with a
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory
names, a value that cannot be cast to that type fails with
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously,
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such
as `2020-01-01 25:00:00`, became the start of that da [...]
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index c41bcbf76ad3..ed8fa99074c4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -594,6 +594,21 @@ object Cast extends QueryErrorsBase {
case _ => false
}
+ /**
+ * Returns true if a LEGACY cast from `from` type to `to` type never fails
at runtime. Among
+ * numeric, boolean and string types, except to CHAR/VARCHAR, it wraps
around or saturates on
+ * overflow, or returns null, instead of failing. Returns false for other
LEGACY casts, some of
+ * which can fail, for example from an interval to a number, from a string
to an interval, and
+ * to CHAR/VARCHAR. The patterns matched reflect the current implementation
in the Cast node and
+ * must stay in sync with it.
+ */
+ def isNoThrowLegacyCast(from: DataType, to: DataType): Boolean = (from, to)
match {
+ case (_: NumericType | BooleanType | _: StringType, _: NumericType |
BooleanType) => true
+ case (_: NumericType | BooleanType | _: StringType, s: StringType) =>
+ StringHelper.isPlainString(s)
+ case _ => false
+ }
+
def resolvableNullability(from: Boolean, to: Boolean): Boolean = !from || to
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 78ccafd6f027..842c8e6fed07 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -56,9 +56,14 @@ object ConstantFolding extends Rule[LogicalPlan] {
case _: Attribute => true
case _: Literal => true
// A cast keeps the eval mode it was analyzed with (for example, in a
view), so check it as
- // well. Keep the session check: some LEGACY casts can still fail, such as
an interval
- // overflow, and this should not start folding them in ANSI sessions.
- case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
+ // well. A TRY cast returns null instead of failing, and so do some LEGACY
casts. Keep the
+ // session check so that this does not start folding casts in ANSI
sessions.
+ case c: Cast if !conf.ansiEnabled => c.evalMode match {
+ case EvalMode.TRY => hasNoSideEffect(c.child)
+ case EvalMode.LEGACY if Cast.isNoThrowLegacyCast(c.child.dataType,
c.dataType) =>
+ hasNoSideEffect(c.child)
+ case _ => false
+ }
case _: NoThrow if e.deterministic => e.children.forall(hasNoSideEffect)
case _ => false
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index 997996d28965..f9f93eb1165e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -342,6 +342,25 @@ class ConstantFoldingSuite extends PlanTest {
}
}
+ test("SPARK-60868: Do not fold the size of CreateArray over a LEGACY cast
that can fail") {
+ val t = LocalRelation($"i".int, $"d".dayTimeInterval())
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ Seq(
+ // These cannot fail: a LEGACY cast as in SPARK-37392, and a TRY cast
that returns null.
+ Cast($"i", StringType) -> true,
+ Cast($"d", ByteType, evalMode = EvalMode.TRY) -> true,
+ // These fail with CAST_OVERFLOW, EXCEED_LIMIT_LENGTH, and
CAST_OVERFLOW in the child.
+ Cast($"d", ByteType) -> false,
+ Cast($"i", VarcharType(2)) -> false,
+ Cast(Cast($"d", ByteType), StringType) -> false).foreach { case (cast,
foldable) =>
+ assert(cast.evalMode != EvalMode.ANSI)
+ val query = t.select(Size(CreateArray(Seq(cast))).as("s")).analyze
+ val expected = if (foldable) t.select(Literal(1).as("s")).analyze else
query
+ comparePlans(Optimize.execute(query), expected)
+ }
+ }
+ }
+
test("SPARK-60849: Constant folding test CreateMap") {
// Building a map deduplicates its keys, so its size can be less than the
number of pairs.
withSQLConf(SQLConf.MAP_KEY_DEDUP_POLICY.key ->
SQLConf.MapKeyDedupPolicy.LAST_WIN.toString) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]