Urmi Mustafi created GOBBLIN-1982:
-------------------------------------
Summary: Show a consistent flowExecutionId btwn Compilation &
Execution
Key: GOBBLIN-1982
URL: https://issues.apache.org/jira/browse/GOBBLIN-1982
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-service
Reporter: Urmi Mustafi
Assignee: Abhishek Tiwari
The problem statement addressed in this issue is to determine a unique ID per
execution that is agreed upon by all hosts, computed before returning any
information back to user (about compilation or execution).
Upon receiving the request for an adhoc flow, the recipient host creates a
flowExecutionId when initializing FlowSpec from config for non-scheduled flows
(see
[code|https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#276]).
This flowExecutionId is returned to the user for tracking the flow status.
This should not change later on.
Scheduled flows are fired upon each host at a different system clock time, so
those ones need a consensus mechanism to coordinate between hosts. During
multiActiveLeaseArbitration we update the flowExecutionId of a DagAction with
an agreed upon value from the database to gain this consistency. However, this
should only be done for scheduled flows before we any information externally
about the flowExecutionId until later.
To address the problems above we
1) skip flowExecutionId replacement for adhoc flows
2) remove a flow compilation and GTE emission before the consensus on
flowExecutionId is removed.
There's no significant impact of removing this check. It will result in
dagActions created for flows that may fail compilation later (after lease
arbitration and before execution). Since we already compile the flow on
accepting it, we are okay with a slight delay in failing a flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
