pratyakshsharma commented on a change in pull request #4925:
URL: https://github.com/apache/hudi/pull/4925#discussion_r816195813
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -379,23 +444,50 @@ private static String resetTarget(Config configuration,
String database, String
* Creates actual HoodieDeltaStreamer objects for every table/topic and does
incremental sync.
*/
public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: "
+ context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
+ if (tableExecutionContexts.isEmpty()) {
+ return;
+ }
+ if (tableExecutionContexts.get(0).getConfig().isFetchFromMultiSources) {
+ while (true) {
Review comment:
Also trying to understand this scenario. Suppose a user has a
combination of target tables to be ingested, say table A and table B. Table A
is bound to multiple sources, but table B depends on a single source. I hope
such combinations of tables are taken care. May be we can add test cases for
such combinations if not done already? @watermelon12138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
