yihua commented on code in PR #10866:
URL: https://github.com/apache/hudi/pull/10866#discussion_r1529083756
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/config/SourceTestConfig.java:
##########
@@ -27,22 +27,22 @@
public class SourceTestConfig {
public static final ConfigProperty<Integer> NUM_SOURCE_PARTITIONS_PROP =
ConfigProperty
- .key("hoodie.deltastreamer.source.test.num_partitions")
+ .key("hoodie.streamer.source.test.num_partitions")
Review Comment:
Let's add `.withAlternatives()` to preserve the old config naming to be
backward compatible and use `STREAMER_CONFIG_PREFIX` and
`DELTA_STREAMER_CONFIG_PREFIX`. Similar for other configs in this class,
`SourceTestConfig`. See `HoodieStreamerConfig.CHECKPOINT_PROVIDER_PATH` for
reference:
```
public static final ConfigProperty<String> CHECKPOINT_PROVIDER_PATH =
ConfigProperty
.key(STREAMER_CONFIG_PREFIX + "checkpoint.provider.path")
.noDefaultValue()
.withAlternatives(DELTA_STREAMER_CONFIG_PREFIX +
"checkpoint.provider.path")
.markAdvanced()
.withDocumentation("The path for providing the checkpoints.");
```
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -266,9 +266,9 @@ public void setupTest() {
protected static void
populateInvalidTableConfigFilePathProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"yyyyMMdd");
- props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/invalid_uber_config.properties");
+ props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"yyyyMMdd");
Review Comment:
For key generator-related configs, the config prefix is renamed from
`hoodie.deltastreamer.keygen.timebased.` to `hoodie.keygen.timebased.` (see
class `TimestampKeyGeneratorConfig`).
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -279,10 +279,10 @@ protected static void
populateAllCommonProps(TypedProperties props, String dfsBa
protected static void populateCommonProps(TypedProperties props, String
dfsBasePath) {
props.setProperty("hoodie.datasource.write.keygenerator.class",
TestHoodieDeltaStreamer.TestGenerator.class.getName());
-
props.setProperty("hoodie.deltastreamer.keygen.timebased.output.dateformat",
"yyyyMMdd");
- props.setProperty("hoodie.deltastreamer.ingestion.tablesToBeIngested",
"short_trip_db.dummy_table_short_trip,uber_db.dummy_table_uber");
-
props.setProperty("hoodie.deltastreamer.ingestion.uber_db.dummy_table_uber.configFile",
dfsBasePath + "/config/uber_config.properties");
-
props.setProperty("hoodie.deltastreamer.ingestion.short_trip_db.dummy_table_short_trip.configFile",
dfsBasePath + "/config/short_trip_uber_config.properties");
+ props.setProperty("hoodie.streamer.keygen.timebased.output.dateformat",
"yyyyMMdd");
Review Comment:
Similar here and other places for key generator configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
