pratyakshsharma commented on a change in pull request #4925:
URL: https://github.com/apache/hudi/pull/4925#discussion_r816193990
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -379,23 +444,50 @@ private static String resetTarget(Config configuration,
String database, String
* Creates actual HoodieDeltaStreamer objects for every table/topic and does
incremental sync.
*/
public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: "
+ context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
+ if (tableExecutionContexts.isEmpty()) {
+ return;
+ }
+ if (tableExecutionContexts.get(0).getConfig().isFetchFromMultiSources) {
+ while (true) {
Review comment:
This essentially makes it an ever running job. I guess we do not want
users to have resources up and running always doing ingestion with no option of
having a one time job. From what I understand you are trying to introduce a
continuous mode here, which is not recommended this way. Let us make it
configurable? cc @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
