viirya commented on code in PR #43694:
URL: https://github.com/apache/spark/pull/43694#discussion_r1385390260
##########
docs/sql-migration-guide.md:
##########
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.0, any read of SQL tables takes into consideration the SQL
configs
`spark.sql.files.ignoreCorruptFiles`/`spark.sql.files.ignoreMissingFiles`
instead of the core config
`spark.files.ignoreCorruptFiles`/`spark.files.ignoreMissingFiles`.
- Since Spark 4.0, `spark.sql.hive.metastore` drops the support of Hive prior
to 2.0.0 as they require JDK 8 that Spark does not support anymore. Users
should migrate to higher versions.
- Since Spark 4.0, `spark.sql.parquet.compression.codec` drops the support of
codec name `lz4raw`, please use `lz4_raw` instead.
+- Since Spark 4.0, when overflowing during casting timestamp to byte/short/int
under non-ansi mode, Spark will return null instead a wrapping value.
Review Comment:
Hmm, I think `TRY` mode currently already return null (because it is treated
like ansi mode). So we can ignore it.
Though I remember non-ansi mode (`LEGACY` mode) returns null for overflowing
that follows Hive.
This is LEGACY mode's document:
> LEGACY: the default evaluation mode, which is compliant to Hive SQL.
I verified Hive's behavior (4.0.0-beta-1,
`hive.strict.timestamp.conversion`: false) now.
I use the same test case from `CastSuiteBase`:
```
0: jdbc:hive2://localhost:10000/> select cast(cast("1900-05-05 18:34:56.1"
as timestamp) as int);
+-------+
| _c0 |
+-------+
| NULL |
+-------+
1 row selected (0.109 seconds)
```
```
0: jdbc:hive2://localhost:10000/> select cast(cast("1900-05-05 18:34:56.1"
as timestamp) as bigint);
+--------------+
| _c0 |
+--------------+
| -2198208304 |
+--------------+
1 row selected (0.126 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
