umustafi commented on code in PR #3846:
URL: https://github.com/apache/gobblin/pull/3846#discussion_r1425850602
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -359,6 +363,11 @@ public void submitFlowToDagManager(FlowSpec flowSpec,
Dag<JobExecutionPlan> jobE
try {
// Send the dag to the DagManager
this.dagManager.get().addDag(jobExecutionPlanDag, true, true);
+
+ // Delete spec from flow catalog for adhoc executions after persisting
it in DagManager
+ if (!flowSpec.getConfig().hasPath(ConfigurationKeys.JOB_SCHEDULE_KEY)) {
+ this.flowCatalog.get().remove(flowSpec.getUri(), new Properties(),
false);
+ }
Review Comment:
Yes, it will. If we've gotten to the point where `addDag` persists the `dag`
to `dagStateStore`, now its responsibility of `dagManager` to execute it. The
multi-active scheduler component ensures persisting to `dagActionStore` so the
event will not be reprocessed through `dagActionChangeMonitor`. The
`dagActionChangeMonitor` only commits the offset after submitting a launch
event to the `Orchestrator` for compiling and completing the `addDag` call. The
new leader's `DagManager` will recover it from `DagStateStore` if the old
leader never completes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
