yihua commented on issue #5442: URL: https://github.com/apache/hudi/issues/5442#issuecomment-1111758104
@mandar-mw In order to deduplicate records within and across commits for INSERT operation, you need to see both of the following configs to be `true` (they are `false` by default): [`hoodie.datasource.write.insert.drop.duplicates`](https://hudi.apache.org/docs/configurations#hoodiedatasourcewriteinsertdropduplicates) and [`hoodie.combine.before.insert`](https://hudi.apache.org/docs/configurations#hoodiecombinebeforeinsert). Let me know if this helps to solve your problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
