[
https://issues.apache.org/jira/browse/GOBBLIN-1974?focusedWorklogId=895563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-895563
]
ASF GitHub Bot logged work on GOBBLIN-1974:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Dec/23 23:29
Start Date: 13/Dec/23 23:29
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3846:
URL: https://github.com/apache/gobblin/pull/3846#discussion_r1425987653
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -299,7 +304,7 @@ public void orchestrate(Spec spec, Properties jobProps,
long triggerTimestampMil
// Depending on if DagManager is present, handle execution
if (this.dagManager.isPresent()) {
- submitFlowToDagManager((FlowSpec) spec, jobExecutionPlanDag);
+ submitFlowToDagManager(flowSpec, jobExecutionPlanDag);
Review Comment:
let's consider dropping support for running GaaS w/o the DagManager. what
are your thoughts @Will-Lo ?
(not as part of this PR, but as a follow-up in the next month or so)
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -342,6 +348,15 @@ public void orchestrate(Spec spec, Properties jobProps,
long triggerTimestampMil
Instrumented.updateTimer(this.flowOrchestrationTimer, System.nanoTime() -
startTime, TimeUnit.NANOSECONDS);
}
+ /*
+ Deletes spec from flowCatalog if it is an adhoc flow (not containing a job
schedule)
+ */
+ private void deleteSpecFromCatalogIfAdhoc(FlowSpec flowSpec) {
Review Comment:
nit: `private` methods like this belong beneath the `public` and `protected`
ones
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -248,7 +253,7 @@ public void orchestrate(Spec spec, Properties jobProps,
long triggerTimestampMil
Instrumented.markMeter(this.flowOrchestrationFailedMeter);
return;
}
- Map<String, String> flowMetadata =
TimingEventUtils.getFlowMetadata((FlowSpec) spec);
+ Map<String, String> flowMetadata =
TimingEventUtils.getFlowMetadata(flowSpec);
Review Comment:
much nicer to get rid of those casts so far separated from the `instanceof`
check!
Issue Time Tracking
-------------------
Worklog Id: (was: 895563)
Time Spent: 1h 10m (was: 1h)
> Ensure Adhoc Flows can be Executed in Multi-active Scheduler state
> ------------------------------------------------------------------
>
> Key: GOBBLIN-1974
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1974
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Adhoc flow executions delete the flow spec from the FlowCatalog after adding
> the spec to the scheduler for the single active scheduler. This makes sense
> in prior context since the single active scheduler has the information and
> the spec is no longer needed. However, in the multi-active case we run into a
> {{No spec found}} error here
> [https://github.com/apache/gobblin/blob/00c60ca6492644652c66905eb384ab145e348bbf/gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java#L263].
> This PR updates the code to delete the spec after persisting it in the adhoc
> execution case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
