danny0405 commented on code in PR #13759:
URL: https://github.com/apache/hudi/pull/13759#discussion_r2308999241
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteFunction.java:
##########
@@ -436,20 +439,22 @@ protected List<WriteStatus> writeRecords(
rowItr, rowData -> recordConverter.convert(rowData,
rowDataBucket.getBucketInfo()));
List<WriteStatus> statuses = writeFunction.write(
- deduplicateRecordsIfNeeded(recordItr), rowDataBucket.getBucketInfo(),
instant);
+ deduplicateRecordsIfNeeded(recordItr,
rowDataBucket.getBucketInfo().getBucketType()), rowDataBucket.getBucketInfo(),
instant);
writeMetrics.endFileFlush();
writeMetrics.increaseNumOfFilesWritten();
return statuses;
}
- protected Iterator<HoodieRecord>
deduplicateRecordsIfNeeded(Iterator<HoodieRecord> records) {
- if (config.get(FlinkOptions.PRE_COMBINE)) {
+ protected Iterator<HoodieRecord>
deduplicateRecordsIfNeeded(Iterator<HoodieRecord> records, BucketType
bucketType) {
+ // do not need deduplication if the merge handle supports deduplicating
+ if (!config.get(FlinkOptions.PRE_COMBINE) ||
OptionsResolver.isMergeHandleSupportDeduplication(config) && bucketType ==
BucketType.UPDATE) {
Review Comment:
looks like the dedup is not just for cow write, then the fix might not be
necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
