qlong commented on code in PR #53295:
URL: https://github.com/apache/spark/pull/53295#discussion_r2590785074
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala:
##########
@@ -269,12 +287,10 @@ class CastWithAnsiOnSuite extends CastSuiteBase with
QueryErrorsBase {
case EvalMode.ANSI =>
assert(checkResult1 ==
DataTypeMismatch(
- errorSubClass = "CAST_WITH_CONF_SUGGESTION",
+ errorSubClass = "CAST_WITHOUT_SUGGESTION",
messageParameters = Map(
"srcType" -> "\"TIMESTAMP\"",
- "targetType" -> "\"BOOLEAN\"",
- "config" -> "\"spark.sql.ansi.enabled\"",
- "configVal" -> "'false'"
+ "targetType" -> "\"BOOLEAN\""
Review Comment:
@allisonwang-db Thanks for review! That is a good question. Yes, setting
`spark.sql.ansi.enabled=false` does work for TIMESTAMP → BOOLEAN if
ansi.enabled is false. I just verified this:
**ANSI mode (Spark 4.0 default)**:
```
[DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION] Cannot resolve "CAST(TIMESTAMP
'2023-01-01 00:00:00' AS BOOLEAN)"
due to data type mismatch: cannot cast "TIMESTAMP" to "BOOLEAN".
```
**ANSI=false**:
```
+------------------------------------------------+
|CAST(TIMESTAMP '2023-01-01 00:00:00' AS BOOLEAN)|
+------------------------------------------------+
| true|
+------------------------------------------------+
```
So users upgrading from Spark 3.5 will have working code that now fails with
no suggestion. I see two possible approaches:
### Option A: Current Implementation (try_cast only)
**Error message for TIMESTAMP → BOOLEAN**:
```
cannot cast "TIMESTAMP" to "BOOLEAN".
```
**Pros**: Aligns with Spark 4.0 direction of ANSI by default
**Cons**: No migration path for users upgrading from old versions where this
cast worked
### Option B: Fallback to Config Suggestion
**Error message for TIMESTAMP → BOOLEAN**:
```
cannot cast "TIMESTAMP" to "BOOLEAN" with ANSI mode on.
If you have to cast TIMESTAMP to BOOLEAN, you can set
"spark.sql.ansi.enabled" as 'false'.
```
**Pros**: Helps users migrate from old versions, provides a working solution
**Cons**: Encourages disabling ANSI mode
---
The JIRA (SPARK-49635) says to remove the ANSI config suggestion but doesn't
explicitly discuss edge cases where try_cast can't help. I chose option A
mainly to remove suggestions but happy to go option B, or other better options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
