[
https://issues.apache.org/jira/browse/GOBBLIN-1997?focusedWorklogId=902741&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-902741
]
ASF GitHub Bot logged work on GOBBLIN-1997:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 30/Jan/24 18:53
Start Date: 30/Jan/24 18:53
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3872:
URL: https://github.com/apache/gobblin/pull/3872#discussion_r1471786633
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java:
##########
@@ -288,6 +267,25 @@ protected void startUp() {
//Do nothing.
}
+ /**
+ * Method to submit a {@link Dag} to the {@link DagManager} and delete adhoc
flowSpecs from the FlowCatalog after
+ * persisting it in the other addDag method called. The DagManager's failure
recovery method ensures the flow will be
+ * executed in the event of downtime.
+ * @param flowSpec
+ * @param dag
+ * @param persist
+ * @param setStatus
+ * @throws IOException
+ */
+ public synchronized void addDag(FlowSpec flowSpec, Dag<JobExecutionPlan>
dag, boolean persist, boolean setStatus)
Review Comment:
Is this function being called? I see the function below is using a different
addDag signature. You probably want to specify a different naming convention
here in that case.
Issue Time Tracking
-------------------
Worklog Id: (was: 902741)
Time Spent: 0.5h (was: 20m)
> Avoid deleting flowSpec too early
> ----------------------------------
>
> Key: GOBBLIN-1997
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1997
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> [https://github.com/apache/gobblin/blob/67c610117c2b757bb15714826e0b0edc3e6b26a5/gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java#L290]
>
> If a non-active host receives and processes the LAUNCH event before the
> leader does, then we encounter a SpecNotFoundException when the leader goes
> to process the event and the execution of the LAUNCH will be missed. This PR
> moves the deletion to the DagManager and does it only after completing addDag
> to persist the event and after checking active status.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
