nsivabalan commented on code in PR #8557:
URL: https://github.com/apache/hudi/pull/8557#discussion_r1176094157
##########
hudi-examples/hudi-examples-spark/src/main/java/org/apache/hudi/examples/spark/HoodieSparkBootstrapExample.java:
##########
@@ -64,7 +64,6 @@ public static void main(String[] args) throws Exception {
.option(DataSourceWriteOptions.PRECOMBINE_FIELD().key(),
preCombineField)
.option(HoodieTableConfig.BASE_FILE_FORMAT.key(),
HoodieFileFormat.ORC.name())
.option(HoodieBootstrapConfig.BASE_PATH.key(), basePath)
- .option(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(),
NonpartitionedKeyGenerator.class.getCanonicalName())
Review Comment:
shouldn't we replace w/ data source key gen class config ?
##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/BootstrapExecutorUtils.java:
##########
@@ -263,10 +262,9 @@ private void initializeTable() throws IOException {
private Pair<String, String> genKeyGenClassAndPartitionColumns() {
String keyGenClass;
- if
(StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key(),
null))) {
- keyGenClass =
props.getString(HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key());
- } else if
(StringUtils.nonEmpty(props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key(),
null))) {
- props.put(HoodieWriteConfig.KEYGENERATOR_TYPE.key(),
props.getString(HoodieBootstrapConfig.KEYGEN_TYPE.key()));
+ if
(StringUtils.nonEmpty(props.getString(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(),
null))) {
Review Comment:
how does backwards compatability work? we are not going to support older
configs? and ask users to set write configs instead?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -591,8 +591,7 @@ class TestHoodieSparkSqlWriter {
HoodieBootstrapConfig.PARALLELISM_VALUE.key -> "4",
DataSourceWriteOptions.OPERATION.key ->
DataSourceWriteOptions.BOOTSTRAP_OPERATION_OPT_VAL,
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition",
- HoodieBootstrapConfig.KEYGEN_CLASS_NAME.key ->
classOf[NonpartitionedKeyGenerator].getCanonicalName)
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "")
Review Comment:
shouldn't we set the key gen class instead of setting partition path config
to empty string ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
