zhedoubushishi commented on a change in pull request #3359:
URL: https://github.com/apache/hudi/pull/3359#discussion_r751780639
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -273,8 +278,26 @@ object DataSourceWriteOptions {
*/
val HIVE_STYLE_PARTITIONING =
KeyGeneratorOptions.HIVE_STYLE_PARTITIONING_ENABLE
- val KEYGENERATOR_CLASS_NAME =
ConfigProperty.key("hoodie.datasource.write.keygenerator.class")
+ /**
+ * Key generator class, that implements will extract the key out of
incoming record
+ *
+ */
+ val keyGeneraterInferFunc =
DataSourceOptionsHelper.scalaFunctionToJavaFunction((p: HoodieConfig) => {
+ if (!p.contains(PARTITIONPATH_FIELD)) {
+ Option.of(classOf[NonpartitionedKeyGenerator].getName)
+ } else {
+ val numOfPartFields = p.getString(PARTITIONPATH_FIELD).split(",").length
+ if (numOfPartFields == 1) {
+ Option.of(classOf[SimpleKeyGenerator].getName)
+ } else {
+ Option.of(classOf[ComplexKeyGenerator].getName)
+ }
+ }
+ })
+ val KEYGENERATOR_CLASS_NAME: ConfigProperty[String] = ConfigProperty
+ .key("hoodie.datasource.write.keygenerator.class")
.defaultValue(classOf[SimpleKeyGenerator].getName)
+ .withInferFunction(keyGeneraterInferFunc)
Review comment:
Yes this is something we can definitely try. But I didn't figure out a
way to implement this, it seems hard to pass a scala function to java code. Let
me know if you have any idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
