kl0u opened a new pull request #11460: [FLINK-16655][FLINK-16657] Introduce embedded executor and use it for Web Submission URL: https://github.com/apache/flink/pull/11460 ## What is the purpose of the change The `EmbeddedExecutor` is part of the [FLIP-85](https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode) effort, which aims at giving the user the option of running his/her `main` method on the cluster, rather than the client. Running the user's main method in "Application Mode" implies: 1) launching a dedicated cluster for the application's jobs 2) running the user's main on the cluster, alongside the Dispatcher The `EmbeddedExecutor` is an `Executor` which is assumed to run alongside the `Dispatcher` of the cluster, on the same machine. Conceptually, it is like the existing `Executors` for session clusters, with the difference that this time there is no need to go through REST as this will be running already on the same machine as the Dispatcher. Given that the Web Submission already runs the user's `main` on the cluster, this PR apart from introducing the `EmbeddedExecutor`, it also wires it to the `JarRunHandler` so that: 1) we can test the executor already 2) we get (partially for now) rid of the pattern of throwing `ProgramAbortException` in order to extract the `JobGraph`. ## Brief change log See the description above. It introduces the `EmbeddedExecutor` with a custom `PipelineExecutorLoader` which is only aware of this executor and is used in the `JarRunHandler`. The handler now, instead of extracting the `JobGraph` itself and submitting to the `Dispatcher`, now it executes the user's main method and let's the new executor do the job graph extraction and submission. To not block any threads on the web frontend, we launch the job in `detached` mode, which is reasonable, as web submission is expected to be a "fire-and-forget" action, and also it is compatible with the current behaviour which does not return the results of operations like `collect()`, `count()` or `print()`. The only difference compared to before is that now, a job with a `print()` will throw an exception (as is the case for detached submission from the CLI), rather than execute the job and silently return nothing. ## Verifying this change It was tested manually on Yarn and local cluster and also it is tested through the updated tests for the `JarRunHandler`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
