[
https://issues.apache.org/jira/browse/GOBBLIN-2111?focusedWorklogId=926398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-926398
]
ASF GitHub Bot logged work on GOBBLIN-2111:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jul/24 19:27
Start Date: 17/Jul/24 19:27
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #4002:
URL: https://github.com/apache/gobblin/pull/4002#discussion_r1681616341
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/LaunchDagTask.java:
##########
@@ -36,4 +43,25 @@ public LaunchDagTask(DagActionStore.DagAction dagAction,
LeaseAttemptStatus.Leas
public <T> T host(DagTaskVisitor<T> visitor) {
return visitor.meet(this);
}
+
+ @Override
+ public final boolean conclude() {
+ try {
+ // Delete adhoc flowSpecs from catalog if the dag was concluded properly
+ if (super.conclude()) {
+ DagManager.DagId dagId =
DagManagerUtils.generateDagId(this.dagAction.getFlowGroup(),
+ this.dagAction.getFlowName(), this.dagAction.getFlowExecutionId());
+ FlowSpec flowSpec =
+
this.dagManagementStateStore.getFlowSpec(FlowSpec.Utils.createFlowSpecUri(dagId.getFlowId()));
+ flowSpec.addProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY,
dagId.getFlowExecutionId());
+ if (!flowSpec.isScheduled()) {
+ dagManagementStateStore.removeFlowSpec(flowSpec);
Review Comment:
we can use `dagManagementStateStore.removeFlowSpec(flowSpec.getUri(), new
Properties(), false);` and then there is no need for that API, right?
I prefer the first one, because that matches FLowCatalog API signature. But
if you decide to use this one then we can delete `void removeFlowSpec(URI uri,
Properties headers, boolean triggerListener)` from DMSS
Issue Time Tracking
-------------------
Worklog Id: (was: 926398)
Time Spent: 1h 50m (was: 1h 40m)
> Delete adhoc flowSpecs from FlowCatalog for DagProcEngine
> ---------------------------------------------------------
>
> Key: GOBBLIN-2111
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2111
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> DagProcessingEngine fails to remove adhoc flow specs from catalog after
> checkpointing the dag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
