[
https://issues.apache.org/jira/browse/GOBBLIN-2111?focusedWorklogId=926420&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-926420
]
ASF GitHub Bot logged work on GOBBLIN-2111:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jul/24 23:02
Start Date: 17/Jul/24 23:02
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #4002:
URL: https://github.com/apache/gobblin/pull/4002#discussion_r1681865265
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/LaunchDagTask.java:
##########
@@ -36,4 +42,25 @@ public LaunchDagTask(DagActionStore.DagAction dagAction,
LeaseAttemptStatus.Leas
public <T> T host(DagTaskVisitor<T> visitor) {
return visitor.meet(this);
}
+
+ @Override
+ public final boolean conclude() {
+ try {
+ // Delete adhoc flowSpecs from catalog if the dag was concluded properly
Review Comment:
You probably want to rename this comment to be "remove adhoc flowspecs after
the adhoc job is launched", to be more accurate to what we want out of the
behavior
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/DagProcessingEngineTest.java:
##########
@@ -54,7 +54,7 @@ public class DagProcessingEngineTest {
private DagManagementTaskStreamImpl dagManagementTaskStream;
private DagTaskStream dagTaskStream;
private DagProcFactory dagProcFactory;
- private static MostlyMySqlDagManagementStateStore dagManagementStateStore;
+ private static MostlyMySqlDagManagementStateStore
testDagManagementStateStore;
Review Comment:
Does this field need to be static? If so please add a comment as to why
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/LaunchDagTask.java:
##########
@@ -36,4 +43,25 @@ public LaunchDagTask(DagActionStore.DagAction dagAction,
LeaseAttemptStatus.Leas
public <T> T host(DagTaskVisitor<T> visitor) {
return visitor.meet(this);
}
+
+ @Override
+ public final boolean conclude() {
+ try {
+ // Delete adhoc flowSpecs from catalog if the dag was concluded properly
+ if (super.conclude()) {
+ DagManager.DagId dagId =
DagManagerUtils.generateDagId(this.dagAction.getFlowGroup(),
+ this.dagAction.getFlowName(), this.dagAction.getFlowExecutionId());
+ FlowSpec flowSpec =
+
this.dagManagementStateStore.getFlowSpec(FlowSpec.Utils.createFlowSpecUri(dagId.getFlowId()));
+ flowSpec.addProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY,
dagId.getFlowExecutionId());
+ if (!flowSpec.isScheduled()) {
+ dagManagementStateStore.removeFlowSpec(flowSpec);
Review Comment:
My take is that I prefer avoiding overloading functions unless absolutely
needed
Issue Time Tracking
-------------------
Worklog Id: (was: 926420)
Time Spent: 2h 40m (was: 2.5h)
> Delete adhoc flowSpecs from FlowCatalog for DagProcEngine
> ---------------------------------------------------------
>
> Key: GOBBLIN-2111
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2111
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> DagProcessingEngine fails to remove adhoc flow specs from catalog after
> checkpointing the dag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
