danny0405 commented on code in PR #7998:
URL: https://github.com/apache/hudi/pull/7998#discussion_r1118428935
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -1063,7 +1063,9 @@ object HoodieSparkSqlWriter {
val recordType = config.getRecordMerger.getRecordType
val shouldCombine = parameters(INSERT_DROP_DUPS.key()).toBoolean ||
- operation.equals(WriteOperationType.UPSERT) ||
+ (operation.equals(WriteOperationType.UPSERT) &&
+ parameters.getOrElse(HoodieWriteConfig.COMBINE_BEFORE_UPSERT.key(),
+ HoodieWriteConfig.COMBINE_BEFORE_UPSERT.defaultValue()).toBoolean) ||
Review Comment:
Actually only MOR table can allow lossening the restriction for
COMBINE_BEFORE_UPSERT, for COW table, the combining of in-mem dataset is a
precondition for the uniqueness of deduplication, Flink does that in this line:
https://github.com/apache/hudi/blob/58e9ce710b405e7653236989912eae49559cd011/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java#L382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
