jonvex commented on code in PR #11943:
URL: https://github.com/apache/hudi/pull/11943#discussion_r1811179861
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala:
##########
@@ -284,8 +286,23 @@ trait ProvidesHoodieConfig extends Logging {
}
}
+ val (recordMergeMode, recordMergeStrategy) = if
(payloadClassName.equals(classOf[ValidateDuplicateKeyPayload].getCanonicalName))
{
+ (RecordMergeMode.CUSTOM.name(),
HoodieRecordMerger.PAYLOAD_BASED_MERGER_STRATEGY_UUID)
+ } else {
+ (RecordMergeMode.EVENT_TIME_ORDERING.name(),
HoodieRecordMerger.DEFAULT_MERGER_STRATEGY_UUID)
+ }
+
+ if
(tableConfig.getPayloadClass.equals(classOf[DefaultHoodieRecordPayload].getCanonicalName)
&&
+
tableConfig.getRecordMergeMode.equals(RecordMergeMode.EVENT_TIME_ORDERING)) {
+ tableConfig.clearValue(HoodieTableConfig.PAYLOAD_CLASS_NAME)
Review Comment:
This is the place where we needed to have the custom handling.
combineOptions takes the tableconfig over the overriding opts. But now we are
always setting the table configs, so the overriding opts will never be used
here. So we discussed and decided that the best solution was just to clear out
the tableconfig here if we think that the user just used the default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
