Github user zivanfi commented on the issue: https://github.com/apache/spark/pull/19250 Hive and Impala introduced the following workaround for timestamp interoperability a long ago: The footer of the Parquet file contains metadata about the library that wrote the file. For Hive and Spark this value is parquet-mr, for Impala it is impala itself, since it has its own implementation. Since Hive and Spark writes using UTC-normalized semantics and Impala writes using timezone-agnostic semantics, we can deduce the used semantics from the writer info. So, when Hive sees a Parquet file written by Impala, it will adjust timestamps to compensate for the difference. Impala has an option ([-convert_legacy_hive_parquet_utc_timestamp](https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_timestamp.html)) to do the same when it sees a Parquet file written by anything else than Impala. There are two problems with this workaround: - Spark does not have a similar logic, so while Impala<->Hive and Hive<->SparkSQL can read each other's timestamps, the Impala->SparkSQL direction does not work. If SparkSQL implemented such a writer-dependent adjustment logic, it would already improve interoperability significantly. - The adjustment depends on the local timezone. This can be problematic if a table contains timestamps of mixed semantics, i.e. some data was written by Impala and some data was written by Hive or SparkSQL. In that case, if the local time used for reading differs from the local time used for writing, some of the timestamps will change and some others won't. Both of these behaviors are okay by themselves, since these are two valid timestamp semantics, but both of them happening on a single table is very counter-intuitive. This would be especially problematic for SparkSQL, since SparkSQL has a session-local timezone setting (Impala and Hive use the server timezone, which tends to remain unchanged). To address both of these issues, we both added the recognition and adjustment of Impala-written timestamps to SparkSQL and also added a table property to record the timezone that should be used for these adjustments, so that mixed table do not lead to unintuitive behaviour any more. We also added this table property to Impala and Hive logic and tested that they can correctly read each other's timestamps. However, our initial commit (which was the first of two commits and was meant to be followed by a follow-up change) got reverted in Spark due to some concerns. Because the table property only provides interoperability if respected by all affected components, we reverted our changes to Hive and Impala as well until we can reach an agreement with Spark. To address the concerns that lead to Reynold to revert our initial commit, Imran made three changes compared to our original proposal: - The adjustment logic was moved to the analyzer. - The writer-specific logic was removed, all timestamps get the same treatment regardless of the component that wrote them. As a result of this, the code became simpler and nicer at the price of a behaviour change: Since Impala-written timestamps are already timezone-agnostic, the user now has to specify UTC in the table property for that table (earlier it didn't matter). It also means that it is no longer possible to fix a table that already has mixed semantics content, since you can not set the table property to UTC as that would make Hive/Spark timestamp wrong and you can't set it to the local timezone either because that would make the Impala timestamps wrong. Although more restricting than our initial proposal, this still seems acceptable, since most existing tables are single-writer only, and once you set the table property all writers will respect it, so after you set the table property you can have mixed-writer tables (they won't become mixed-semantics due to the table pr operty). - The adjustment logic was made file-format-agnostic so that it does not only apply to Parquet but to any kinds of tables. However, we then realized that this will lead to further interoperability problems, thereby we would like to stick with a Parquet-specific approach as we originally proposed to avoid making the situation worse.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org