nsivabalan commented on code in PR #5205:
URL: https://github.com/apache/hudi/pull/5205#discussion_r845185260
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -173,6 +175,29 @@ object HoodieWriterUtils {
}
}
+ /**
+ * Detects conflicts between datasourceKeyGen and existing table
configuration keyGen
+ */
+ def validateKeyGeneratorConfig(datasourceKeyGen: String, tableConfig:
HoodieConfig): Unit = {
+ val diffConfigs = StringBuilder.newBuilder
+
+ if (null != tableConfig) {
+ val tableConfigKeyGen =
tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
+ if (null != tableConfigKeyGen && null != datasourceKeyGen) {
+ val nonPartitionedTableConfig =
tableConfigKeyGen.equals(classOf[NonpartitionedKeyGenerator].getCanonicalName)
+ val simpleKeyDataSourceConfig =
datasourceKeyGen.equals(classOf[SimpleKeyGenerator].getCanonicalName)
+ if (nonPartitionedTableConfig && simpleKeyDataSourceConfig) {
+
diffConfigs.append(s"KeyGenerator:\t$datasourceKeyGen\t$tableConfigKeyGen\n")
Review Comment:
yes, I also prefer to check all possible combination of switches.
I have created a follow up task
https://issues.apache.org/jira/browse/HUDI-3820
@rkkalluri : there are 2 work items in there. a: adding validations and
tests for switching between diff key gens. b: with insert_overwrite_table
operation, we should not do the validation and over-write table config if key
gen is changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
