Sarfaraz-214 commented on code in PR #5071:
URL: https://github.com/apache/hudi/pull/5071#discussion_r1584145214
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:
##########
@@ -425,15 +432,29 @@ private static String resetTarget(Config configuration,
String database, String
* Creates actual HoodieDeltaStreamer objects for every table/topic and does
incremental sync.
*/
public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: "
+ context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
- }
- }
+ ExecutorService executorService =
Executors.newFixedThreadPool(tableExecutionContexts.size());
+ tableExecutionContexts.forEach(context -> {
+ executorService.execute(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ HoodieDeltaStreamer streamerInstance = new
HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties()));
+ int iteration = 1;
+ while (iteration <= context.getConfig().continuousIterations) {
+ if
(streamerInstance.getDeltaSyncService().getDeltaSync().isDataAvailableForIngestion())
{
Review Comment:
Hi @pratyakshsharma
What is the implication of not having the **if** condition?
I removed it locally because of this use-case: If I kill an existing
MultiTableStreamer job and relaunch it again and upon relaunch the jobs doesn't
see any new messages in Kafka to consume then it would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
