MaxGekk commented on issue #25678: [SPARK-28973][SQL] Add `TimeType` and support `java.time.LocalTime` as its external type. URL: https://github.com/apache/spark/pull/25678#issuecomment-530251260 > Why do we need a new data type? @rxin , I see the following reasons to add new type `TIME`: 1. To be compliant with SQL standard. 2. To maintain feature parity with PostgreSQL which supports the type. And for example, we have to comment the `TIME` constructor in our tests for now: https://github.com/apache/spark/blob/3a4afce96c6840431ed45280742f9e969be19639/sql/core/src/test/resources/sql-tests/inputs/pgSQL/date.sql#L353 3. To be able to read `TIME` columns and especially write such columns from/to Parquet files (and other formats as well). Let's image, you have a parquet file with `TIME` columns (https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time) and wants to update the file, but you cannot do that by Spark for now because Spark cannot write `TIME` values. 4. Similar reason as above but related to JDBC is to work with modern RDBMS that highly likely support `TIME` type. If an user need to write `TIME` column, she/he cannot do that by Spark without UDFs. 5. To make Catalyst's time-related types more consistent. For now you can make `TIMESTAMP` from `DATE` by casting the former one which is actually `DATE` + `00:00:00.000000` + `TimeZone` = `TIMESTAMP`. You cannot add arbitrary time to date to get a timestamp. If we support `TIME` type, this will give users more flexibility and makes Spark's type system more consistent: `DATE` + `TIME` + `TimeZone` = `TIMESTAMP`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
