[
https://issues.apache.org/jira/browse/SPARK-27790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-27790:
-------------------------------
Description:
Spark has an INTERVAL data type, but it is “broken”:
# It cannot be persisted
# It is not comparable because it crosses the month day line. That is there is
no telling whether “1 Month 1 Day” is equal to “1 Month 1 Day” since not all
months have the same number of days.
I propose here to introduce the two flavours of INTERVAL as described in the
ANSI SQL Standard and deprecate the Sparks interval type.
* ANSI describes two non overlapping “classes”:
* YEAR-MONTH,
* DAY-SECOND ranges
* Members within each class can be compared and sorted.
* Supports datetime arithmetic
* Can be persisted.
The old and new flavors of INTERVAL can coexist until Spark INTERVAL is
eventually retired. Also any semantic “breakage” can be controlled via legacy
config settings.
was:
SQL standard defines 2 interval types:
# year-month interval contains a YEAR field or a MONTH field or both
# day-time interval contains DAY, HOUR, MINUTE, and SECOND (possibly fraction
of seconds)
Need to add 2 new internal types YearMonthIntervalType and DayTimeIntervalType,
support operations defined by SQL standard as well as INTERVAL literals.
The java.time.Period and java.time.Duration can be supported as external type
for YearMonthIntervalType and DayTimeIntervalType.
> Support ANSI SQL INTERVAL types
> -------------------------------
>
> Key: SPARK-27790
> URL: https://issues.apache.org/jira/browse/SPARK-27790
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Maxim Gekk
> Assignee: Apache Spark
> Priority: Major
>
> Spark has an INTERVAL data type, but it is “broken”:
> # It cannot be persisted
> # It is not comparable because it crosses the month day line. That is there
> is no telling whether “1 Month 1 Day” is equal to “1 Month 1 Day” since not
> all months have the same number of days.
> I propose here to introduce the two flavours of INTERVAL as described in the
> ANSI SQL Standard and deprecate the Sparks interval type.
> * ANSI describes two non overlapping “classes”:
> * YEAR-MONTH,
> * DAY-SECOND ranges
> * Members within each class can be compared and sorted.
> * Supports datetime arithmetic
> * Can be persisted.
> The old and new flavors of INTERVAL can coexist until Spark INTERVAL is
> eventually retired. Also any semantic “breakage” can be controlled via legacy
> config settings.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
