AshinGau opened a new issue, #11003:
URL: https://github.com/apache/hudi/issues/11003

   **Describe the problem you faced**
   I am a committer of [Doris](https://github.com/apache/doris). When I use 
Doris to read the parquet file created by hudi, I find that the output of 
timestamp is decreased by 8 hours. Then I use other tools(arrow, trino, spark) 
to check the result, it seems that different system parse different time zone 
of timestamp type:
   
![img_v3_029s_b170ce06-8440-4e8d-b37b-1962ea92b7bg](https://github.com/apache/hudi/assets/19337507/27e39867-8117-4b2c-a725-cb9a3eb20544)
   The results of arrow, Doris, trino are the same, while the results of hudi, 
spark-shell are increased 8 hours.
   
   **To Reproduce**
   Spark 3.3 + Hudi 0.14.1
   1. create hudi table
   ```
   create table hudi_evolution_mor(
     id int,
     name string,
     create_time timestamp,
     price double,
     ts bigint,
     fs_col string) using hudi
   options(
     type = 'mor',
     primaryKey = 'id'
   )
   ```
   2. insert data
   ```
   insert into hudi_evolution_mor values
   (1, 'name1', timestamp'2023-09-17 13:14:35.142', 1.01, 1001, '2023-09-17'),
   (2, 'name2', timestamp'2024-03-10 15:17:21.4172', 2.02, 1002, '2024-03-10');
   ```
   
   **Expected behavior**
   Maybe it's right to output `2023-09-17 13:14:35.142`, because string like 
`timestamp'2023-09-17 13:14:35.142'` is inserted. However the parquet file 
create by hudi indicates that the timestamp type should be adjusted to UTC:
   
![img_v3_029s_8f01c4d1-f8d3-47cf-93ea-8205fbdc8f2g](https://github.com/apache/hudi/assets/19337507/3434ee04-8f40-42e0-a4b8-8ac6023c581d)
   
![img_v3_029s_3e622d05-90a9-4912-a973-8c37bdcf331g](https://github.com/apache/hudi/assets/19337507/cfa7892b-5260-42a8-8599-465f790cee72)
   The meta data shows that the timestamp is relative to UTC0, not the local 
time zone, according to the [definition of 
isAdjustedToUTC=true](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#instant-semantics-timestamps-normalized-to-utc)
   
![image](https://github.com/apache/hudi/assets/19337507/1f1139a6-f58f-4026-8d9c-d828dc81779b)
   
   **Environment Description**
   
   * Hudi version : 0.14.1
   
   * Spark version : 3.3.2
   
   * Hive version : 3.1.1
   
   * Hadoop version : 3.2.2
   
   * Storage (HDFS/S3/GCS..) :
   
   * Running on Docker? (no) :
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to