umustafi commented on code in PR #3863:
URL: https://github.com/apache/gobblin/pull/3863#discussion_r1465694903
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java:
##########
@@ -281,6 +282,7 @@ static void
addJobStatusToStateStore(org.apache.gobblin.configuration.State jobS
stateStore.put(storeName, tableName, jobStatus);
if (isNewStateTransitionToFinal(jobStatus, states)) {
eventProducer.emitObservabilityEvent(jobStatus);
+ eventBus.post(new JobStatusEvent(jobStatus));
Review Comment:
why is this change here? can u add a comment to explain who the listener is
here of the job status event?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceGuiceModule.java:
##########
@@ -173,6 +180,12 @@ public void configure(Binder binder) {
binder.bind(FlowTriggerHandler.class);
}
+ binder.bind(DagManagement.class).to(NewDagManager.class);
+
binder.bind(DagManagementStateStore.class).to(InMemoryDagManagementStateStore.class);
+ binder.bind(DagProcFactory.class);
+ binder.bind(DagProcessingEngine.class);
+ binder.bind(DagTaskStream.class);
Review Comment:
these should be optionally bound through the config
`isDagProcessingEngineEnabled `
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java:
##########
@@ -250,7 +259,11 @@ private void handleDagAction(DagActionStore.DagAction
dagAction, boolean isStart
throw new RuntimeException(String.format("Received LAUNCH dagAction
while not in multi-active scheduler "
+ "mode for flowAction: %s", dagAction));
}
- submitFlowToDagManagerHelper(dagAction, isStartup);
+ if (dagProcessingEngineEnabled) {
+ this.dagTaskStream.addDagAction(dagAction);
+ } else {
+ submitFlowToDagManagerHelper(dagAction, isStartup);
+ }
Review Comment:
in this PR we should still have it in "test" mode so if dagProcessingEngine
is labeled we add some logging around the code paths it follows and we can
trace it but not necessarily change the path yet. What do you think?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -210,18 +206,6 @@ public
GobblinServiceJobScheduler(@Named(InjectionNames.SERVICE_NAME) String ser
}
}
- public GobblinServiceJobScheduler(String serviceName, Config config,
FlowStatusGenerator flowStatusGenerator,
- Optional<HelixManager> helixManager, Optional<FlowCatalog> flowCatalog,
TopologyCatalog topologyCatalog,
- DagManager dagManager, Optional<UserQuotaManager> quotaManager,
SchedulerService schedulerService,
- Optional<Logger> log, boolean isWarmStandbyEnabled, Optional
<FlowTriggerHandler> flowTriggerHandler,
- SharedFlowMetricsSingleton sharedFlowMetricsSingleton)
- throws Exception {
- this(serviceName, config, helixManager, flowCatalog,
- new Orchestrator(config, topologyCatalog, dagManager, log,
flowStatusGenerator, flowTriggerHandler,
- sharedFlowMetricsSingleton, flowCatalog),
- schedulerService, quotaManager, log, isWarmStandbyEnabled,
flowTriggerHandler);
- }
-
Review Comment:
why did this get removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]