GJL commented on a change in pull request #9791: [FLINK-14248][runtime] Let
LazyFromSourcesSchedulingStrategy restart terminated tasks
URL: https://github.com/apache/flink/pull/9791#discussion_r330574514
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##########
@@ -78,19 +81,22 @@ public void startScheduling() {
deploymentOptions.put(schedulingVertex.getId(), option);
}
-
allocateSlotsAndDeployExecutionVertexIds(getAllVerticesFromTopology());
+ allocateSlotsAndDeployExecutionVertices(
+
getSchedulingExecutionVertices(getAllVerticesFromTopology()),
+ IS_IN_CREATED_STATE);
}
@Override
- public void restartTasks(Set<ExecutionVertexID> verticesToRestart) {
+ public void restartTasks(final Set<ExecutionVertexID>
verticesToRestart) {
+ final Set<SchedulingExecutionVertex> verticesToSchedule =
getSchedulingExecutionVertices(verticesToRestart);
+
// increase counter of the dataset first
- verticesToRestart
+ verticesToSchedule
.stream()
- .map(schedulingTopology::getVertexOrThrow)
.flatMap(vertex ->
vertex.getProducedResultPartitions().stream())
.forEach(inputConstraintChecker::resetSchedulingResultPartition);
- allocateSlotsAndDeployExecutionVertexIds(verticesToRestart);
+ allocateSlotsAndDeployExecutionVertices(verticesToSchedule,
IS_IN_TERMINAL_STATE);
Review comment:
> A vertex is transitioned to `CREATED` only when it is reset. This only
happens in `allocateSlotsAndDeploy` in `DefaultScheduler`.
> The vertex versions will change in this case.
> And these vertices will be filtered out to restart.
I understand that we reset the vertices in `allocateSlotsAndDeploy`.
However, if we reset the vertices before invoking
`SchedulingStrategy#restartTasks()`, it looks to me that we don't need to
distinguish in `allocateSlotsAndDeployExecutionVertices` between a restarted
task and regular deployments. That is, we could always filter on `CREATED`
state. I am looking for reasons why this is a bad idea.
Moreover, it seems that we could replace `IS_IN_TERMINAL_STATE` with `x ->
true`, and the algorithm would still work.
The bottom line is, we have to decide on a contract. Before invoking
`restartTasks()`, should all vertices be in `CREATED` or in a terminal state?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
