gaoyunhaii opened a new pull request #16633:
URL: https://github.com/apache/flink/pull/16633
## What is the purpose of the change
This PR avoids that for a checkpoint, if an operator has used
`UnionListState`, we would expected to include the state of all the subtasks to
avoid possible issues. We achieve this by abort checkpoints if we found that
1. Part of tasks of a vertex has turn into FINISHED, and remaining tasks
have reported `UnionListState`.
2. All of the tasks of a vertex are running, but some tasks have called
operators.finish() and others are not.
Then
1. If for a vertex it does not wait for the final checkpoint (due to the
possible optimization), it might come across both case 1 and case 2. After all
the tasks of this vertex are finished, the checkpoint would recover.
2. If for a vertex it waits for the final checkpoint, then the finished
tasks would be blocked before waiting the final checkpoints since checkpoints
are aborted. Until all the subtasks reach this position, the checkpoint would
be successful, then one checkpoint would unblock all these tasks.
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a
persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
This PR is verified by the added UT.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
