MaxGekk opened a new pull request #25678: [WIP][SPARK-28973][SQL] Add `TimeType` and support `java.time.LocalTime` as its external type. URL: https://github.com/apache/spark/pull/25678 ### What changes were proposed in this pull request? Proposed new type for Catalyst's type system to represent local time. According to the SQL standard, a value of data type `TIME` comprises values of the datetime fields HOUR, MINUTE and SECOND. It is always a valid time of day. - `HOUR` - hour within day, between 00 and 23 - `MINUTE` - minute within hour, between 00 and 59 - `SECOND` - second and possibly fraction of a second within minute. Valid range is 0-59.999999 Internally, the `TIME` type is implemented as Catalyst `TimeType` and stores a number of microseconds since `00:00:00`. The java class `java.time.LocalTime` was supported as the external type for `TimeType`. So, instances of `java.time.LocalTime` can be parallelized and converted to values of `TimeType`, and collected back to instances of `LocalTime` as well. Spark also accepts literals of `TimeType` with values of `LocalTime`. Created an encoder that serializes instances of the `java.time.LocalTime` classto the internal representation of nullable Catalyst's `TimeType`. ### Why are the changes needed? * To maintain feature parity with PostgreSQL which supports the `TIME` type and its constructor `make_time`. The former function is commented in `date.sql` at the moment: https://github.com/apache/spark/blob/3a4afce96c6840431ed45280742f9e969be19639/sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql#L353 * To be compliant with the SQL standard ### Does this PR introduce any user-facing change? The PR extends existing functionality. So, users can parallelize instances of the `java.time.LocalTime` class and collect them back: ```Scala scala> val ds = Seq(java.time.LocalTime.of(20, 38, 10)).toDS ds: org.apache.spark.sql.Dataset[java.time.LocalTime] = [value: time] scala> ds.collect res5: Array[java.time.LocalTime] = Array(20:38:10) ``` ### How was this patch tested? - Added a few tests to `CatalystTypeConvertersSuite` to check conversion from/to `java.time.LocalTime`. - Checking row encoding by new tests in `RowEncoderSuite`. - Making literals of `TimeType` is tested in `LiteralExpressionSuite` - Check collecting by `DatasetSuite` and `JavaDatasetSuite`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
