XComp opened a new pull request #18627: URL: https://github.com/apache/flink/pull/18627
This PR is related to [FLIP-194](https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore) and based on PR #18626 . It's labeled as a draft as long as PR #18626 isn't merged into `master`. ## What is the purpose of the change This change introduces a `JobManagerRunner` implementation taking care of the cleanup of a otherwise finished job. ## Brief change log * makes certain `Testing*` implementations more general to make testing of the new functionality easier * Moves extraction of `state.checkpoints.num-retained` from `SchedulerUtils` into `DefaultCompletedCheckpointStoreUtils` (which is renamed into `CompletedCheckpointStoreUtils`) to align with the new use case for this functionality * Makes `ApplicationStatus` <-> `JobStatus` mapping (partially) symmetric * Introduces `CheckpointResourcesCleanupRunner` as a new `JobManagerRunner` implementation (+ correspondng factory class to support testing) and integrates it into the `Dispatcher` * Reorganizes Dispatcher tests moving certain tests out of `DispatcherTest` into `DispatcherResourceCleanupTest` ## Verifying this change This change added tests and can be verified as follows: * ApplicationMode is tests in `ApplicationDispatcheerBootstrapITCase` * General Dispatcher is tested in `DispatcherFailoverITCase` * `DispatcherTest.testJobCleanupWithoutRecoveredJobGraph` testing that the job is cleaned up through a `CheckpointResourcesCleanupRunner` rather than initializing a new `JobMaster` * `CheckpointResourcesCleanupRunnerTest` was added to cover new class `CheckpointResourcesCleanupRunner` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs / JavaDocs * JavaDoc is provided * A `JobResultStore` section is added under `Deployment / High Availability` * `JobResultStore` is added to the glossary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
