Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19250
IIUC, using the `parquet.timezone-adjustment` table property requires
changing the writer. e.g. Impala creates a table and Hive wants to write data
to it, then Hive needs to write timezone-agnostic timestamp instead of UTC
timestamp. And vice versa.
I think a better solution is the Parquet-specific approach. Impala keeps
writing timezone-agnostic timestamp, and Spark/Hive keep writing UTC timestamp.
At the read path, we should detect who wrote the parquet file and do timestamp
adjustment based on this information(can also have a config to protect this
behavior).
cc @rxin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
