Zhujun-Vungle opened a new issue #639: About global dedup, find some commit who 
keep inflight and still generate parquet file and fail dedup
URL: https://github.com/apache/incubator-hudi/issues/639
 
 
   Hi,
   Taking investigation on hudi, currently facing a issue when trying hudi 
global dedup 
   ## ENV 
   data source: kafka
   etl engine: spark streaming which simply ingest data per 10 minutes and 
write to s3 with global dedup enable
   data sink: s3
   viewer: hive table which create like 
https://cwiki.apache.org/confluence/display/HUDI/Registering+sample+dataset+to+Hive+via+beeline
   ```
   ROW FORMAT SERDE
   'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
   STORED AS INPUTFORMAT
   'com.uber.hoodie.hadoop.HoodieInputFormat'
   OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
   ```
   
   ## HUDI config
   
   
   ```
   kafkaBlock{
   // per 10 minutes data as a dataframe
   df => 
         spark
           .sql("select date_format(timestamp,'yyyy-MM-dd_HH') as ts,* from t1")
           .dropDuplicates("event_id")
           .toDF()
           .write
           .format("com.uber.hoodie")
           .mode(SaveMode.Append)
           .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "event_id")
           .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "ts")
           .option(DataSourceWriteOptions.OPERATION_OPT_KEY, 
DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL)
           .option(HoodieWriteConfig.TABLE_NAME, "table1")
           // global dedup
           .option(HoodieIndexConfig.INDEX_TYPE_PROP, 
HoodieIndex.IndexType.GLOBAL_BLOOM.name)
           //   before write optimization
           .option("hoodie.combine.before.insert", "true")
           .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,
             "timestamp")
           .option(DataSourceWriteOptions.INSERT_DROP_DUPS_OPT_KEY, "true")
          .save(dst)
   }
   ```
   
   ## phenomenon
   Order by last update time 
   ![Screen Shot 2019-04-16 at 4 22 00 
PM](https://user-images.githubusercontent.com/22140275/56193814-195c8280-6064-11e9-8f09-c45f2ebc98b6.png)
   
   The spark streaming running log, no delay at all
   ![Screen Shot 2019-04-16 at 4 22 53 
PM](https://user-images.githubusercontent.com/22140275/56193934-60e30e80-6064-11e9-990e-64cff58361b0.png)
   
   
   ## question 
   After running pipeline for a while, I query the hive table(running in spark 
sql```select event_id,count(*) from hudi_table group by 1 having COUNT(*) > 1 
order by 1 desc limit 10```), I got duplicate records. The duplicate records 
are coming from the inflight commit.
   So
   1. Did I missing some hudi config in writing pipeline?
   2. Why the commit is still inflight but parquet file can be generated?
   3. In hudi, did the commit finish and then spark .save finish so there is no 
asynchronous problem?
   4. Is there any reader side config issue with hive output format ? since 
it's org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   
   Regards

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to