MaxGekk commented on issue #23811: [SPARK-26902][SQL] Support java.time.Instant as an external type of TimestampType URL: https://github.com/apache/spark/pull/23811#issuecomment-579967121 > it means almost nobody will be using the new type right? From my point of view, this is debatable statement. Java 8 is 6 years old already. I could guess significant amount of modern apps including Spark apps is written on top of Java 8 time API. I do think users will look for how to parallelize Java 8 time-related values to Spark. Maybe we should highlight in Spark SQL docs more clearly how do that by using the flag. > My point is that it's not great when you have a feature that's almost never on and mostly just be dead code. Actually the `spark.sql.datetime.java8API.enabled` config controls only output types (from Spark). Regarding input types - Spark accepts both legacy and new one `java.time.Instant`. If an app or UDF are written using Java 8 API, user will look for ways to take `Instant` from Spark as well. There is no so much "dead" code in the changes. > Is there a plan to transition over to the new type? Spark 3.0 is going to introduce Java 8 time classes so far. In this release, we could keep both with old returned types by default. I would switch on Java 8 time API by default in the next release - Spark 3.1 or 3.2. From another side, the major release is good time for switching since Java 8 time API is mature enough. > Wouldn't that plan involve breaking a lot of user code? I don't have statistics on hands. Switching on Java 8 time API by default, definitely will break: 1. UDFs and 2. apps that collects results from Spark. In the first case, we could try to detect input types of UDF, and maybe avoid failures by passing legacy types. but in the case of collecting datasets from Spark, it depends on users apps. In any case, the code can be easily fixed by converting, for instance `java.time.Instant` to `java.sqlTimestamp` via: `java.sql.Timestamp.from(instant)`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
