[
https://issues.apache.org/jira/browse/GOBBLIN-2151?focusedWorklogId=934116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-934116
]
ASF GitHub Bot logged work on GOBBLIN-2151:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 10/Sep/24 23:46
Start Date: 10/Sep/24 23:46
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #4050:
URL: https://github.com/apache/gobblin/pull/4050#discussion_r1752939273
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/utils/FlowCompilationValidationHelper.java:
##########
@@ -187,9 +194,59 @@ public Optional<Dag<JobExecutionPlan>>
validateAndHandleConcurrentExecution(Conf
* @param allowConcurrentExecution
* @return true if the {@link FlowSpec} allows concurrent executions or if
no other instance of the flow is currently RUNNING.
*/
- private boolean isExecutionPermitted(FlowStatusGenerator
flowStatusGenerator, String flowGroup, String flowName,
- boolean allowConcurrentExecution, long flowExecutionId) {
- return allowConcurrentExecution ||
!flowStatusGenerator.isFlowRunning(flowName, flowGroup, flowExecutionId);
+ private boolean isExecutionPermitted(String flowGroup, String flowName,
boolean allowConcurrentExecution)
+ throws IOException {
+ return allowConcurrentExecution || !isFlowRunning(flowGroup, flowName,
dagManagementStateStore);
+ }
+
+ /**
+ * Returns true if any previous execution for the flow determined by the
provided flowGroup, flowName is running.
+ * We ignore the execution that has the provided flowExecutionId. We also
ignore the flows that are running beyond
Review Comment:
what about re-attemps to process a launch event that was only partially
successful? until processing succeeds fully, a lease is available for the same
`DagAction`.
from `LaunchDagProc::initialize`:
```
protected Optional<Dag<JobExecutionPlan>>
initialize(DagManagementStateStore dagManagementStateStore)
throws IOException {
try {
FlowSpec flowSpec =
dagManagementStateStore.getFlowSpec(FlowSpec.Utils.createFlowSpecUri(getDagId().getFlowId()));
flowSpec.addProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY,
getDagId().getFlowExecutionId());
Optional<Dag<JobExecutionPlan>> dag =
this.flowCompilationValidationHelper.createExecutionPlanIfValid(flowSpec).toJavaUtil();
if (dag.isPresent()) {
dagManagementStateStore.addDag(dag.get());
}
return dag;
...
```
if the prior attempt got to `dagManagementStateStore.addDag` and then
failed, wouldn't the subsequent exec never get past
`flowCompilationValidationHelper.createExecutionPlanIfValid`, unless this
method (called within) agrees to skip past that same (currently validating)
flowExecId?
Issue Time Tracking
-------------------
Worklog Id: (was: 934116)
Time Spent: 2h 20m (was: 2h 10m)
> ignore flows that are running beyond job start and flow finish deadline
> -----------------------------------------------------------------------
>
> Key: GOBBLIN-2151
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2151
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
