rxin commented on issue #25678: [SPARK-28973][SQL] Add `TimeType` and support `java.time.LocalTime` as its external type. URL: https://github.com/apache/spark/pull/25678#issuecomment-531054262 I haven't had a chance but I'd push back against most of the functionalities in the umbrella ticket (Postgres compatibility). While I think it's a good idea in general to align with some existing standard when a functionality exists, a blanket "implement some other database's functionality" is a dangerous umbrella, because the context and use cases are very different between Spark and "some other databases". A few reasons: 1. Different systems evolve differently, and systems that have been around longer have a lot of legacy that even the maintainers of such systems regret. For example, maybe nobody (or worse, somebody but less than 0.01% so you can't get rid of it) uses a particular feature. 2. Database systems have a very different architecture from Spark's more modern big data world. One constraint is that SQL is often the only way in, and SQL is the only way out. UDFs are very difficult to do. Think recursive subquery as an example ... in Spark you can easily accomplish a similar thing by just using the programming language's capabilities, which is far more flexible than SQL's recursion. 3. There's a cost to adding some new code or feature to the codebase. In the case of a new data type, it has high overhead to the end user and library developer as well, because now they need to handle this data type in their code. It's one thing if a lot of users are asking for this functionality; it's a very different thing if the whole motivation is to add it because we want to have parity with some database that we don't even know if any of the Spark use cases need.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
