Gengliang Wang created SPARK-36227:
--------------------------------------
Summary: Remove TimestampNTZ type support in Spark 3.2
Key: SPARK-36227
URL: https://issues.apache.org/jira/browse/SPARK-36227
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.2.0
Reporter: Gengliang Wang
Assignee: Gengliang Wang
As of now, there are some blockers for delivering the TimestampNTZ project in
Spark 3.2:
# In the Hive Thrift server, both TimestampType and TimestampNTZType are mapped
to the same timestamp type, which can cause confusion for users.
# For the Parquet data source, the new written TimestampNTZType Parquet columns
will be read as TimestampType in old Spark releases. Also, we need to decide
the merge schema for files mixed with TimestampType and TimestampNTZ type.
# The type coercion rules for TimestampNTZType are incomplete. For example,
what should the data type of the in clause "IN(Timestamp'2020-01-01 00:00:00',
TimestampNtz'2020-01-01 00:00:00') be.
# It is tricky to support TimestampNTZType in JSON/CSV data readers. We need to
avoid regressions as possible as we can.
There are 10 days left for the expected 3.2 RC date. So, I propose to release
the TimestampNTZ type in Spark 3.3 instead of Spark 3.2. So that we have enough
time to make considerate designs for the issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
