Github user zivanfi commented on the issue:
https://github.com/apache/spark/pull/19250
The interoperability issue is that Impala follows timezone-agnostic
timestamp semantics as mandated by the SQL standard, while SparkSQL follows
UTC-normalized semantics instead (which is not SQL-compliant). Please see the
[design
doc|https://docs.google.com/document/d/1XmyVjr3eOJiNFjVeSnmjIU60Hq-XiZB03pgi3r1razM/edit]
for details.
The long-term solution is using separate SQL types for different timestamps
semantics indeed as you suggest. However, until we get there we would like to
have a short-term solution that fixes timestamps that are _already written_.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
