Zhujun-Vungle opened a new issue #639: About global dedup, find some commit who keep inflight and still generate parquet file and fail dedup URL: https://github.com/apache/incubator-hudi/issues/639 Hi, Taking investigation on hudi, currently facing a issue when trying hudi global dedup ## ENV data source: kafka etl engine: spark streaming which simply ingest data per 10 minutes and write to s3 with global dedup enable data sink: s3 viewer: hive table which create like https://cwiki.apache.org/confluence/display/HUDI/Registering+sample+dataset+to+Hive+via+beeline ``` ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'com.uber.hoodie.hadoop.HoodieInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet. ``` ## HUDI config ``` kafkaBlock{ // per 10 minutes data as a dataframe df => spark .sql("select date_format(timestamp,'yyyy-MM-dd_HH') as ts,* from t1") .dropDuplicates("event_id") .toDF() .write .format("com.uber.hoodie") .mode(SaveMode.Append) .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "event_id") .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "ts") .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL) .option(HoodieWriteConfig.TABLE_NAME, "table1") // global dedup .option(HoodieIndexConfig.INDEX_TYPE_PROP, HoodieIndex.IndexType.GLOBAL_BLOOM.name) // before write optimization .option("hoodie.combine.before.insert", "true") .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "timestamp") .option(DataSourceWriteOptions.INSERT_DROP_DUPS_OPT_KEY, "true") .save(dst) } ``` ## phenomenon Order by last update time  The spark streaming running log, no delay at all  ## question After running pipeline for a while, I query the hive table(running in spark sql```select event_id,count(*) from hudi_table group by 1 having COUNT(*) > 1 order by 1 desc limit 10```), I got duplicate records. So 1. Did I missing some hudi config here? 2. Why the commit is stil inflight but parquet file can be generated? 3. Did the commit finish and then spark save finish so there is no asynchronous problem? Regards
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
