Zoltan Ivanfi has posted comments on this change. Change subject: IMPALA-2716: Hive/Impala incompatibility for timestamp data in Parquet ......................................................................
Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/5939/3//COMMIT_MSG Commit Message: Line 23: global flag --prevent_parquet_mr_zone_adjustment is set to true. > Not sure if the name of this flag was already decided upon, but imo, it's n It is also important that it is only set on new tables and that it is set to UTC. I would suggest --set_parquet_mr_int96_write_zone_to_utc_on_new_tables http://gerrit.cloudera.org:8080/#/c/5939/3/be/src/service/impala-server.cc File be/src/service/impala-server.cc: Line 123: "Impala, Hive, and Spark to not apply timestamp timezone adjustments for parquet " > Mention that this property also makes Hive write in UTC. Hive always writes in UTC (that's the problem). It makes Hive write as if it was located in UTC instead of the actual local timezone, which makes the written value match the original value. I think this effect is so complex that we can not describe it in a few lines. I would replace the last sentence with "This changes the behavior of recent versions of Hive and Spark as well. The writing of timestamps is affected in Hive and Spark but not in Impala. The reading of timestamps that were written by Hive, Spark, or any other parquet-mr writer is affected in Hive, Spark and Impala. You can find details in the documentation." -- To view, visit http://gerrit.cloudera.org:8080/5939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3f24525ef45a2814f476bdee76655b30081079d6 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges <[email protected]> Gerrit-Reviewer: Alex Behm <[email protected]> Gerrit-Reviewer: Attila Jeges <[email protected]> Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Taras Bobrovytsky <[email protected]> Gerrit-Reviewer: Zoltan Ivanfi <[email protected]> Gerrit-HasComments: Yes
