[I] [SUPPORT]Flink writes MOR table, both RO table and RT table read nothing by hive [hudi]

Tue, 09 Jan 2024 05:00:56 -0800 


nicholasxu opened a new issue, #10465:
URL: https://github.com/apache/hudi/issues/10465

   **Describe the problem you faced**
   
   I use Flink write HUDI MOR table, and Flink read table normally, while RO 
table and RT table read nothing by hive
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. Create hive catalog as follows:
   
   CREATE CATALOG hudi_hive_catalog
     WITH (
       'type'='hudi',
       'catalog.path' = 'cosn://bigdata-xxx/user/hive/warehouse', 
       'hive.conf.dir' = '/usr/local/service/hive/conf',
       'mode'='hms',
       'table.external' = 'true', 
       'default-database' = 'hudi_default' 
     );
   
   2. Use catalog and create table  as follows:
   
   use CATALOG hudi_hive_catalog;
   
   CREATE TABLE t1(
     mid BIGINT PRIMARY KEY NOT ENFORCED, 
     uuid VARCHAR(20),
     name VARCHAR(10),
     age INT,
     ts BIGINT,
     part INT
   )
   PARTITIONED BY (part)
   WITH (
     'connector' = 'hudi',
     'path' = 'cosn://bigdata-xxx/user/hive/warehouse/hudi_default.db/t1',
     'table.type' = 'MERGE_ON_READ', 
     'hive_sync.enable' = 'true',  
     'hive_sync.mode' = 'hms',        
     'hive_sync.metastore.uris' = 'thrift://xxx:9083'
   )
   
   3.Insert some data by Flink
   INSERT INTO t1
   VALUES
   (1,'334e26e9-8355-45cc-97c6-c31daf0df330','nick', 18,1695159649087,20230108),
   (2,'334e26e9-8355-45cc-97c6-c31daf0df330','jack', 18,1695159649087,20230109);
   
   4.Read data by Flink and get right records
   SELECT * FROM t1;
   
![image](https://github.com/apache/hudi/assets/12593964/30cb73e2-22a1-414f-8035-b52f5ad9e6ac)
   
   5. Use 'select *' reading data by Hive and get nothing
   select * from t1_rt;
   
![image](https://github.com/apache/hudi/assets/12593964/6ba664d5-f9af-4a80-bd1c-a4b258b1ad30)
   select * from t1_ro; 
   
![image](https://github.com/apache/hudi/assets/12593964/da205ace-79a8-4a98-98f2-5d37ccf91c56)
   
   7. Read data with 'order by clause' by Hive and get right results
   select * from t1_rt order by mid;
   
![image](https://github.com/apache/hudi/assets/12593964/0965d50c-f390-43b0-be11-e5e3a21f0d3c)
   select * from t1_ro order by mid;
   
![image](https://github.com/apache/hudi/assets/12593964/eeab5e22-8ef9-4112-8a4e-2389489a2373)
   
   8.Huid files on cos：
   
![image](https://github.com/apache/hudi/assets/12593964/8871477d-7713-499f-ac47-2b8644f46220)
   
   9.Test COW table is ok
   
   **Expected behavior**
   Reading nothing from RO table  may be OK, because it only has a log file and 
without parquet base files, 
   but reading nothing from RT table is confused,  your help is appreciative.
   
   
   **Environment Description**
   
   * Hudi version :0.14.1
   
   * Spark version :3.2.2
   
   * Hive version :3.1.3
   
   * Hadoop version :3.2.2
   
   * Storage (HDFS/S3/GCS..) :COS on Tencent Cloud
   
   * Running on Docker? (yes/no) :
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to