[
https://issues.apache.org/jira/browse/GOBBLIN-1982?focusedWorklogId=899032&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-899032
]
ASF GitHub Bot logged work on GOBBLIN-1982:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/24 17:09
Start Date: 10/Jan/24 17:09
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3854:
URL: https://github.com/apache/gobblin/pull/3854#discussion_r1447678638
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MultiActiveLeaseArbiter.java:
##########
@@ -53,10 +53,14 @@ public interface MultiActiveLeaseArbiter {
* @param flowAction uniquely identifies the flow and the present action
upon it
* @param eventTimeMillis is the time this flow action was triggered
* @param isReminderEvent true if the flow action event we're checking on is
a reminder event
+ * @param skipFlowExecutionIdReplacement if true then does not replace the
flowExecutionId in the flowAction returned
Review Comment:
renamed to the latter and updated tests
Issue Time Tracking
-------------------
Worklog Id: (was: 899032)
Time Spent: 1.5h (was: 1h 20m)
> Show a consistent flowExecutionId btwn Compilation & Execution
> ---------------------------------------------------------------
>
> Key: GOBBLIN-1982
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1982
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The problem statement addressed in this issue is to determine a unique ID per
> execution that is agreed upon by all hosts, computed before returning any
> information back to user (about compilation or execution).
> Upon receiving the request for an adhoc flow, the recipient host creates a
> flowExecutionId when initializing FlowSpec from config for non-scheduled
> flows (see
> [code|https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#276]).
> This flowExecutionId is returned to the user for tracking the flow status.
> This should not change later on.
> Scheduled flows are fired upon each host at a different system clock time, so
> those ones need a consensus mechanism to coordinate between hosts. During
> multiActiveLeaseArbitration we update the flowExecutionId of a DagAction with
> an agreed upon value from the database to gain this consistency. However,
> this should only be done for scheduled flows before we any information
> externally about the flowExecutionId until later.
> To address the problems above we
> 1) skip flowExecutionId replacement for adhoc flows
> 2) remove a flow compilation and GTE emission before the consensus on
> flowExecutionId is removed.
> There's no significant impact of removing this check. It will result in
> dagActions created for flows that may fail compilation later (after lease
> arbitration and before execution). Since we already compile the flow on
> accepting it, we are okay with a slight delay in failing a flow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
