[ https://issues.apache.org/jira/browse/HIVE-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368895#comment-17368895 ]
Nikhil Gupta commented on HIVE-25104: ------------------------------------- I have a list of issues which we can club together: (These are after hive 3.1 release): # HIVE-25093: date_format() UDF is returning output in UTC time zone only (Ashish Sharma, reviewed by Adesh Rao, Nikhil Gupta, Sankar Hariappan) # HIVE-25104: Backward incompatible timestamp serialization in Parquet for certain timezones (Stamatis Zampetakis, reviewed by Jesus Camacho Rodriguez) # HIVE-24113: NPE in GenericUDFToUnixTimeStamp (#1460) (Raj Kumar Singh, reviewed by Zoltan Haindrich and Laszlo Pinter) # HIVE-24074: Incorrect handling of timestamp in Parquet/Avro when written in certain time zones in versions before Hive 3.x (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) # HIVE-22840: Race condition in formatters of TimestampColumnVector and DateColumnVector (Shubham Chaurasia, reviewed by Jesus Camacho Rodriguez) # HIVE-22589: Add storage support for ProlepticCalendar in ORC, Parquet, and Avro (Jesus Camacho Rodriguez, reviewed by David Lavati, L<C3><A1>szl<C3><B3> Bodor, Prasanth Jayachandran) # HIVE-22405: Add ColumnVector support for ProlepticCalendar (L<C3><A1>szl<C3><B3> Bodor via Owen O'Malley, Jesus Camacho Rodriguez) # HIVE-22331: unix_timestamp without argument returns timestamp in millisecond instead of second (Naresh P R, reviewed Jesus Camacho Rodriguez) # HIVE-22170: from_unixtime and unix_timestamp should use user session time zone (Jesus Camacho Rodriguez, reviewed by Vineet Garg) # HIVE-21729: Arrow serializer sometimes shifts timestamp by one second (Shubham Chaurasia, reviewed by Sankar Hariappan) # HIVE-21291: Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time (Karen Coppage, reviewed by Jesus Camacho Rodriguez) > Backward incompatible timestamp serialization in Parquet for certain timezones > ------------------------------------------------------------------------------ > > Key: HIVE-25104 > URL: https://issues.apache.org/jira/browse/HIVE-25104 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers > Affects Versions: 3.1.0 > Reporter: Stamatis Zampetakis > Assignee: Stamatis Zampetakis > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > HIVE-12192, HIVE-20007 changed the way that timestamp computations are > performed and to some extend how timestamps are serialized and deserialized > in files (Parquet, Avro). > In versions that include HIVE-12192 or HIVE-20007 the serialization in > Parquet files is not backwards compatible. In other words writing timestamps > with a version of Hive that includes HIVE-12192/HIVE-20007 and reading them > with another (not including the previous issues) may lead to different > results depending on the default timezone of the system. > Consider the following scenario where the default system timezone is set to > US/Pacific. > At apache/master commit 37f13b02dff94e310d77febd60f93d5a205254d3 > {code:sql} > CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET > LOCATION '/tmp/hiveexttbl/employee'; > INSERT INTO employee VALUES (1, '1880-01-01 00:00:00'); > INSERT INTO employee VALUES (2, '1884-01-01 00:00:00'); > INSERT INTO employee VALUES (3, '1990-01-01 00:00:00'); > SELECT * FROM employee; > {code} > |1|1880-01-01 00:00:00| > |2|1884-01-01 00:00:00| > |3|1990-01-01 00:00:00| > At apache/branch-2.3 commit 324f9faf12d4b91a9359391810cb3312c004d356 > {code:sql} > CREATE EXTERNAL TABLE employee(eid INT,birth timestamp) STORED AS PARQUET > LOCATION '/tmp/hiveexttbl/employee'; > SELECT * FROM employee; > {code} > |1|1879-12-31 23:52:58| > |2|1884-01-01 00:00:00| > |3|1990-01-01 00:00:00| > The timestamp for {{eid=1}} in branch-2.3 is different from the one in master. -- This message was sent by Atlassian Jira (v8.3.4#803005)