watermelon12138 commented on a change in pull request #4925:
URL: https://github.com/apache/hudi/pull/4925#discussion_r816449044
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -379,23 +444,50 @@ private static String resetTarget(Config configuration,
String database, String
* Creates actual HoodieDeltaStreamer objects for every table/topic and does
incremental sync.
*/
public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: "
+ context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
+ if (tableExecutionContexts.isEmpty()) {
+ return;
+ }
+ if (tableExecutionContexts.get(0).getConfig().isFetchFromMultiSources) {
+ while (true) {
Review comment:
Thank you for your good advice. First, I added the parameter
allowContinuousWhenMultipleSources to control whether to use the continuous
mode in the case of multiple sources. Second, In the original logic, I only
allowed the sink table to be unique. However, I accept your suggestion.
Currently, I support multiple sink tables, and each sink table can be bound to
multiple sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
