echauchot opened a new pull request #13040: URL: https://github.com/apache/flink/pull/13040
## What is the purpose of the change Back pressure the checkpointing system when there are too many checkpoints to clean. See design doc here: https://docs.google.com/document/d/1q0y0aWlJMoUWNW7jjsM8uWfHsy2dM6YmmcmhpQzgLMA/edit?usp=sharing Please see comments in the implementation plan section for an explanation of the choices made (classes responsibilities, interfaces, thresholds ...) ## Brief change log - CheckpointStores are now responsible for checkpoints to clean count. This count is incremented when an asynchronous execution of checkpoint discard is submitted and decremented when the discard is finished (either fail or success) - CheckpointRequestDecider stops choosing new checkpoint trigger requests while there are too many checkpoints to clean. When the number of checkpoints to clean becomes below the threshold, CheckpointRequestDecider starts to pick up checkpoint requests again. ## Verifying this change This change added tests and can be verified as follows: stop and resume checkpoint trigger request pick up by CheckpointRequestDecider. I did not make a IT test for endurance (verify there is no OOM error) when there is a high frequency of checkpoints cleaning requests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes checkpointing) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not applicable) R: @ifndef-SleePy @rkhachatryan . Thanks guys ! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
