Maxim Gekk created SPARK-27528:
----------------------------------
Summary: Use Parquet logical type TIMESTAMP_MICROS by default
Key: SPARK-27528
URL: https://issues.apache.org/jira/browse/SPARK-27528
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.1
Reporter: Maxim Gekk
Currently, Spark uses INT96 type for timestamps written to parquet files. To
store Catalyst's Timestamp values as INT96, Spark converts microseconds since
epoch to nanoseconds in Julian calendar. This conversion is not necessary if
Spark saves timestamps as Parquet TIMESTAMP_MICROS logical type. The ticket
aims to switch on TIMESTAMP_MICROS from INT96 in write by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
