This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ab9cc0870e96 [SPARK-49635][SQL] Remove ANSI config suggestion in CAST
error messages
ab9cc0870e96 is described below
commit ab9cc0870e96cbf93d0859d2a8b48195eea5f65d
Author: Qiegang Long <[email protected]>
AuthorDate: Tue Dec 9 09:33:47 2025 -0800
[SPARK-49635][SQL] Remove ANSI config suggestion in CAST error messages
### What changes were proposed in this pull request?
This PR improves CAST error messages to provide suggestions based on
whether casts work in ANSI mode, non-ANSI mode, or try_cast():
1. Suggest try_cast() when the cast is valid but fails in ANSI mode and
try_cast() supports it
2. Suggest disabling ANSI mode when the cast works in non-ANSI mode but
try_cast() doesn't support it
3. Provide no suggestion when the cast is invalid in all modes
### Why are the changes needed?
In Spark 4.0.0, ANSI mode is now enabled by default. The previous error
messages suggested users disable ANSI mode when encountering cast failures,
which goes against the direction of the project. However, completely removing
these suggestions would leave users without guidance when migrating code from
Spark 3.x that relied on non-ANSI cast behavior.
### Does this PR introduce _any_ user-facing change?
Yes. Error messages for CAST operations that fail in ANSI mode now suggest
using `try_cast()` instead of disabling ANSI mode, but only when `try_cast()`
actually supports that cast operation.
**Before:**
```
cannot cast "ARRAY<INT>" to "ARRAY<BINARY>" with ANSI mode on.
If you have to cast ARRAY<INT> to ARRAY<BINARY>, you can set
"spark.sql.ansi.enabled" as 'false'.
```
**After (for casts supported by try_cast):**
```
cannot cast "ARRAY<INT>" to "ARRAY<BINARY>".
To convert values from "ARRAY<INT>" to "ARRAY<BINARY>", you can use the
functions `try_cast` instead.
```
**After (suggests disabling ANSI when try_cast doesn't support it):**
```
cannot cast "TIMESTAMP" to "BOOLEAN" with ANSI mode on.
This cast is not allowed in ANSI mode but works when
"spark.sql.ansi.enabled" is 'false'.
```
**After (for casts not supported by try_cast):**
```
cannot cast "INT" to "BINARY".
```
### How was this patch tested?
- Added new test case for complex type casts (Array[Int] to Array[Binary])
that
demonstrates try_cast suggestion
- Added test case for TIMESTAMP → BOOLEAN cast that demonstrates config
suggestion
- Updated existing test expectations in CastWithAnsiOnSuite to reflect the
new
behavior
- All Cast-related unit tests pass
- Verified no regression in legacy (ANSI off) mode
- Manually tested with spark-shell to verify error messages
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53295 from qlong/SPARK-49635-remove-ansi-suggestion.
Authored-by: Qiegang Long <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>
---
.../spark/sql/catalyst/expressions/Cast.scala | 19 +++++++++++++++--
.../catalyst/expressions/CastWithAnsiOnSuite.scala | 24 ++++++++++++++++++++++
2 files changed, 41 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 1162a5394221..3e9a07feb8c6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -505,6 +505,10 @@ object Cast extends QueryErrorsBase {
"config" -> toSQLConf(fallbackConf.get._1),
"configVal" -> toSQLValue(fallbackConf.get._2, StringType)))
+ case _ if fallbackConf.isEmpty && Cast.canTryCast(from, to) =>
+ // Suggest try_cast for valid casts that fail in ANSI mode
+ withFunSuggest("try_cast")
+
case _ =>
DataTypeMismatch(
errorSubClass = "CAST_WITHOUT_SUGGESTION",
@@ -588,8 +592,19 @@ case class Cast(
Some(SQLConf.STORE_ASSIGNMENT_POLICY.key ->
SQLConf.StoreAssignmentPolicy.LEGACY.toString))
} else {
- Cast.typeCheckFailureMessage(child.dataType, dataType,
- Some(SQLConf.ANSI_ENABLED.key -> "false"))
+ // Check if there's a config workaround for this cast failure:
+ // - If canTryCast supports this cast, pass None here and let
typeCheckFailureMessage
+ // suggest try_cast (which is more user-friendly than disabling ANSI
mode)
+ // - If canTryCast doesn't support it BUT the cast works in non-ANSI
mode,
+ // suggest disabling ANSI mode as a migration path
+ // - Otherwise, pass None and let typeCheckFailureMessage decide
+ val fallbackConf = if (!Cast.canTryCast(child.dataType, dataType) &&
+ Cast.canCast(child.dataType, dataType)) {
+ Some(SQLConf.ANSI_ENABLED.key -> "false")
+ } else {
+ None
+ }
+ Cast.typeCheckFailureMessage(child.dataType, dataType, fallbackConf)
}
case EvalMode.TRY =>
Cast.typeCheckFailureMessage(child.dataType, dataType, None)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
index b03d97abd900..b76aec6d6ce0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
@@ -211,6 +211,30 @@ class CastWithAnsiOnSuite extends CastSuiteBase with
QueryErrorsBase {
}
}
+ test("SPARK-49635: suggest try_cast for complex type casts") {
+ // Array[Int] to Array[Binary]: canTryCast=true (uses canCast),
canAnsiCast=false
+ // Should suggest try_cast, not config
+ val arrayIntType = ArrayType(IntegerType, containsNull = false)
+ val arrayBinaryType = ArrayType(BinaryType, containsNull = false)
+ val arrayIntLiteral = Literal.create(Seq(1, 2, 3), arrayIntType)
+
+ val arrayResult = cast(arrayIntLiteral,
arrayBinaryType).checkInputDataTypes()
+ evalMode match {
+ case EvalMode.ANSI =>
+ assert(arrayResult ==
+ DataTypeMismatch(
+ errorSubClass = "CAST_WITH_FUNC_SUGGESTION",
+ messageParameters = Map(
+ "srcType" -> toSQLType(arrayIntType),
+ "targetType" -> toSQLType(arrayBinaryType),
+ "functionNames" -> "`try_cast`"
+ )
+ )
+ )
+ case _ =>
+ }
+ }
+
test("ANSI mode: disallow variant cast to non-nullable types") {
// Array
val variantVal = new VariantVal(Array[Byte](12, 3), Array[Byte](1, 0, 0))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
