[
https://issues.apache.org/jira/browse/GOBBLIN-2151?focusedWorklogId=934110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-934110
]
ASF GitHub Bot logged work on GOBBLIN-2151:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 10/Sep/24 23:05
Start Date: 10/Sep/24 23:05
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #4050:
URL: https://github.com/apache/gobblin/pull/4050#discussion_r1752916478
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/utils/FlowCompilationValidationHelper.java:
##########
@@ -187,9 +194,59 @@ public Optional<Dag<JobExecutionPlan>>
validateAndHandleConcurrentExecution(Conf
* @param allowConcurrentExecution
* @return true if the {@link FlowSpec} allows concurrent executions or if
no other instance of the flow is currently RUNNING.
*/
- private boolean isExecutionPermitted(FlowStatusGenerator
flowStatusGenerator, String flowGroup, String flowName,
- boolean allowConcurrentExecution, long flowExecutionId) {
- return allowConcurrentExecution ||
!flowStatusGenerator.isFlowRunning(flowName, flowGroup, flowExecutionId);
+ private boolean isExecutionPermitted(String flowGroup, String flowName,
boolean allowConcurrentExecution)
+ throws IOException {
+ return allowConcurrentExecution || !isFlowRunning(flowGroup, flowName,
dagManagementStateStore);
+ }
+
+ /**
+ * Returns true if any previous execution for the flow determined by the
provided flowGroup, flowName is running.
+ * We ignore the execution that has the provided flowExecutionId. We also
ignore the flows that are running beyond
Review Comment:
We need to ignore the current flowExecutionId because other hosts may have
already come to a decision with the current processed flow status. Since in the
multileader world, the hosts can become desyncronized it is important that they
come to the same decision, otherwise one host can set a dag as RUNNING then
another host which is running late will check the current status, and decide
that the dag should FAIL due to already being set as RUNNING.
Issue Time Tracking
-------------------
Worklog Id: (was: 934110)
Time Spent: 1.5h (was: 1h 20m)
> ignore flows that are running beyond job start and flow finish deadline
> -----------------------------------------------------------------------
>
> Key: GOBBLIN-2151
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2151
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
