[GitHub] [hudi] so-lazy commented on issue #2338: [SUPPORT] MOR table found duplicate and process so slowly

Mon, 28 Dec 2020 00:56:03 -0800 


so-lazy commented on issue #2338:
URL: https://github.com/apache/hudi/issues/2338#issuecomment-751638099


   I run `select id,count(1) from table_ro group by id having count(1) > 1` got 
this
   
![image](https://user-images.githubusercontent.com/56884416/103202016-4b8f4000-492c-11eb-8ca6-9c6d0bf63f5a.png)
   
   then i query data by id got this
   
![image](https://user-images.githubusercontent.com/56884416/103202036-53e77b00-492c-11eb-8be4-645e874b344f.png)
   
   then i go to check the parquet file
   `val df = 
spark.read.parquet("/hudi/2020-12/c147c22c-7e86-4ae9-bc77-e2a2a52cc897-0_0-230-113454_20201215205055.parquet")
   df.createOrReplaceTempView("table")
   spark.sql("select 
_hoodie_commit_time,_hoodie_commit_seqno,_hoodie_record_key,_hoodie_partition_path,_hoodie_file_name,id
 from table WHERE id = 338409469").show`
   
![image](https://user-images.githubusercontent.com/56884416/103202457-6e6e2400-492d-11eb-82b2-d74740077e12.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to