[
https://issues.apache.org/jira/browse/GOBBLIN-1982?focusedWorklogId=899030&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899030
]
ASF GitHub Bot logged work on GOBBLIN-1982:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/24 16:52
Start Date: 10/Jan/24 16:52
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3854:
URL: https://github.com/apache/gobblin/pull/3854#discussion_r1447657951
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -463,7 +461,7 @@ private void deleteFromExecutor(Spec spec, Properties
headers) {
Deletes spec from flowCatalog if it is an adhoc flow (not containing a job
schedule)
*/
private void deleteSpecFromCatalogIfAdhoc(FlowSpec flowSpec) {
- if (!flowSpec.getConfig().hasPath(ConfigurationKeys.JOB_SCHEDULE_KEY)) {
+ if (!flowSpec.isScheduled()) {
Review Comment:
Yes, I went to add this util method thinking it would be useful and found it
already exists
Issue Time Tracking
-------------------
Worklog Id: (was: 899030)
Time Spent: 1h 20m (was: 1h 10m)
> Show a consistent flowExecutionId btwn Compilation & Execution
> ---------------------------------------------------------------
>
> Key: GOBBLIN-1982
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1982
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The problem statement addressed in this issue is to determine a unique ID per
> execution that is agreed upon by all hosts, computed before returning any
> information back to user (about compilation or execution).
> Upon receiving the request for an adhoc flow, the recipient host creates a
> flowExecutionId when initializing FlowSpec from config for non-scheduled
> flows (see
> [code|https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#276]).
> This flowExecutionId is returned to the user for tracking the flow status.
> This should not change later on.
> Scheduled flows are fired upon each host at a different system clock time, so
> those ones need a consensus mechanism to coordinate between hosts. During
> multiActiveLeaseArbitration we update the flowExecutionId of a DagAction with
> an agreed upon value from the database to gain this consistency. However,
> this should only be done for scheduled flows before we any information
> externally about the flowExecutionId until later.
> To address the problems above we
> 1) skip flowExecutionId replacement for adhoc flows
> 2) remove a flow compilation and GTE emission before the consensus on
> flowExecutionId is removed.
> There's no significant impact of removing this check. It will result in
> dagActions created for flows that may fail compilation later (after lease
> arbitration and before execution). Since we already compile the flow on
> accepting it, we are okay with a slight delay in failing a flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
