nsivabalan commented on a change in pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#discussion_r789367919
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -378,16 +383,23 @@ private static String resetTarget(Config configuration,
String database, String
/**
* Creates actual HoodieDeltaStreamer objects for every table/topic and does
incremental sync.
*/
- public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: "
+ context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
- }
- }
+ public void sync() throws InterruptedException {
+ ExecutorService executorService =
Executors.newFixedThreadPool(tableExecutionContexts.size());
+ tableExecutionContexts.forEach(context -> {
+ executorService.execute(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
+ successTables.add(Helpers.getTableWithDatabase(context));
+ } catch (Exception e) {
+ logger.error("error while running MultiTableDeltaStreamer for
table: " + context.getTableName(), e);
+ failedTables.add(Helpers.getTableWithDatabase(context));
+ }
+ }
+ });
+ });
+ executorService.shutdown();
Review comment:
should we add awaitTermination here ? we can't proceed w/ next batch
until all tables in current batch is completed right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
