phet commented on code in PR #3893:
URL: https://github.com/apache/gobblin/pull/3893#discussion_r1520783083
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceGuiceModule.java:
##########
@@ -248,7 +262,12 @@ public void configure(Binder binder) {
if (serviceConfig.isWarmStandbyEnabled()) {
binder.bind(SpecStoreChangeMonitor.class).toProvider(SpecStoreChangeMonitorFactory.class).in(Singleton.class);
-
binder.bind(DagActionStoreChangeMonitor.class).toProvider(DagActionStoreChangeMonitorFactory.class).in(Singleton.class);
+ if (serviceConfig.isDagProcessingEngineEnabled()) {
+ binder.bind(DagActionStoreChangeMonitor.class)
+
.toProvider(DagProcEngineEnabledDagActionStoreChangeMonitorFactory.class).in(Singleton.class);
Review Comment:
did we not update the name of the factory to align w/
the`DagManagementDagActionStoreChangeMonitor` it creates?
if not, shall we now bring them into alignment?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceManager.java:
##########
@@ -192,6 +193,8 @@ public class GobblinServiceManager implements
ApplicationLauncher, StandardMetri
@Getter
@VisibleForTesting
public DagManager dagManager;
+ @Inject
+ DagManagement dagManagement;
Review Comment:
I'd deem `DagProcEngine` the most approximate replacement for `DagManager`.
don't we want one of those here?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -208,18 +214,20 @@ public
GobblinServiceJobScheduler(@Named(InjectionNames.SERVICE_NAME) String ser
metricContext.register(this.totalAddSpecTimeNanos);
metricContext.register(this.numJobsScheduledDuringStartup);
}
+ this.dagProcessingEngineEnabled = ConfigUtils.getBoolean(config,
ServiceConfigKeys.DAG_PROCESSING_ENGINE_ENABLED, false);
Review Comment:
I keep seeing classes looking into config to find out this value. is there
really no way to have guice calculate/read it once and then pass the value to
each of these other classes it's already involved in initializing?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProc.java:
##########
@@ -45,26 +63,147 @@ public class LaunchDagProc extends
DagProc<Optional<Dag<JobExecutionPlan>>, Opti
public LaunchDagProc(LaunchDagTask launchDagTask) {
this.launchDagTask = launchDagTask;
this.orchestrationDelayCounter = new AtomicLong(0);
- ContextAwareGauge<Long> orchestrationDelayMetric =
this.metricContext.newContextAwareGauge
+ ContextAwareGauge<Long> orchestrationDelayMetric =
metricContext.newContextAwareGauge
(ServiceMetricNames.FLOW_ORCHESTRATION_DELAY,
orchestrationDelayCounter::get);
- this.metricContext.register(orchestrationDelayMetric);
+ metricContext.register(orchestrationDelayMetric);
}
@Override
protected Optional<Dag<JobExecutionPlan>> initialize(DagManagementStateStore
dagManagementStateStore)
throws IOException {
- throw new UnsupportedOperationException("Not yet implemented");
+ return dagManagementStateStore.getDag(this.launchDagTask.getDagId());
}
@Override
protected Optional<Dag<JobExecutionPlan>> act(DagManagementStateStore
dagManagementStateStore, Optional<Dag<JobExecutionPlan>> dag)
throws IOException {
- throw new UnsupportedOperationException("Not yet implemented");
+ if (!dag.isPresent()) {
+ log.warn("No dag with id " + this.launchDagTask.getDagId() + " found to
launch");
+ return Optional.empty();
+ }
+ initializeDag(dag.get(), dagManagementStateStore);
+ return dag;
Review Comment:
there's already a lot to this PR w/ just the guice mods. I glanced a few
moments ago and thought I noticed you setting aside the `LaunchDagProc` impl
for a separate PR... didn't I?
I strongly endorse separating the two! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
