Maxim Gekk created SPARK-31076:
----------------------------------
Summary: Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp
via local date-time
Key: SPARK-31076
URL: https://issues.apache.org/jira/browse/SPARK-31076
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Maxim Gekk
By default, collect() returns java.sql.Timestamp/Date instances with offsets
derived from internal values of Catalyst's TIMESTAMP/DATE that store
microseconds since the epoch. The conversion from internal values to
java.sql.Timestamp/Date based on Proleptic Gregorian calendar but converting
the resulted values before 1582 year to strings produces timestamp/date string
in Julian calendar. For example:
{code}
scala> sql("select date '1100-10-10'").collect()
res1: Array[org.apache.spark.sql.Row] = Array([1100-10-03])
{code}
This can be fixed if internal Catalyst's values are converted to local
date-time in Gregorian calendar, and construct local date-time from the
resulted year, month, ..., seconds in Julian calendar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
