[I] [SUPPORT] spark-sql query results in duplicates, duplicates span multiple files in the same file group [hudi]

Tue, 27 Feb 2024 18:33:15 -0800 


ligou525 opened a new issue, #10773:
URL: https://github.com/apache/hudi/issues/10773

   **Describe the problem you faced**
   query hudi table by spark-sql, duplicates occure:
   MOR table:
   
![5a4f9a4a7301d61c4102e44c6a285b3d](https://github.com/apache/hudi/assets/31082145/969686ce-2f31-4913-8282-f34d48dc22af)
   file structure:
   
![image](https://github.com/apache/hudi/assets/31082145/47097eac-a435-433d-8f78-0547210c03f2)
   hoodie properties:
   
![image](https://github.com/apache/hudi/assets/31082145/afbdb9b9-c9ac-4909-9b3b-53b639fcfcc4)
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. use hudi-java-client to write data to hudi table with upsert
   2. query data by spark-sql
   
   **Expected behavior**
   
   no duplicates
   
   **Environment Description**
   
   * Hudi version : 0.12.2
   
   * Spark version : 3.2
   
   * Hive version : 3.1.3
   
   * Hadoop version : 3.2.3
   
   * Storage (HDFS/S3/GCS..) : hdfs
   
   * Running on Docker? (yes/no) : no
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to