zentol commented on a change in pull request #8804: [FLINK-12883][WIP][runtime]
Add elaborated partition release logic
URL: https://github.com/apache/flink/pull/8804#discussion_r299014120
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java
##########
@@ -1605,36 +1629,9 @@ public boolean updateState(TaskExecutionState state) {
if (attempt != null) {
try {
- Map<String, Accumulator<?, ?>> accumulators;
-
- switch (state.getExecutionState()) {
- case RUNNING:
- return
attempt.switchToRunning();
-
- case FINISHED:
- // this deserialization is
exception-free
- accumulators =
deserializeAccumulators(state);
-
attempt.markFinished(accumulators, state.getIOMetrics());
- return true;
-
- case CANCELED:
- // this deserialization is
exception-free
- accumulators =
deserializeAccumulators(state);
-
attempt.completeCancelling(accumulators, state.getIOMetrics());
- return true;
-
- case FAILED:
- // this deserialization is
exception-free
- accumulators =
deserializeAccumulators(state);
-
attempt.markFailed(state.getError(userClassLoader), accumulators,
state.getIOMetrics());
- return true;
-
- default:
- // we mark as failed and return
false, which triggers the TaskManager
- // to remove the task
- attempt.fail(new
Exception("TaskManager sent illegal state update: " +
state.getExecutionState()));
- return false;
- }
+ final boolean stateUpdated =
updateStateInternal(state, attempt);
+ maybeReleasePartitions(state, attempt);
Review comment:
We should either take `stateUpdated` into account here, or check the actual
execution state opposed to what is in the task state update.
If you have a vertex in a CANCELING state, which then receives a FINISHED
state updated, the vertex completes the canceling and switches to CANCELED. In
this situation we will already issue release calls for all partitions this
vertex has produced.
The strategy however would still consider this vertex as finished, and
potentially release preceding partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
