Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/542#issuecomment-87840336
I know that the PR is touching many components. I'll try to split up my
work into smaller parts.
In this case I didn't expect in the beginning that I need to change so many
things.
The PR is mainly about adding support for executing a Flink job on YARN in
a "fire and forget" fashion.
Therefore, I needed to make some changes to the YARN client.
In the previous big change to YARN, I added support for a "detached" yarn
session. So that you can tell the Flink Yarn Client to start Flink on YARN
without connecting to the AM afterwards. Users have to manage such a yarn
session using other tools afterwards (for example `yarn application -kill` to
stop it)
This change brings this feature even further to support single flink jobs
being submitted to YARN. But since the Yarn client doesn't connect to the AM
once Flink has been started, there is no way to tell the AM to stop Flink on
YARN again.
In this change, I add a new Akka message for the ApplicationMaster `case
class StopAMAfterJob(jobId:JobID)`. The message tells the AM to monitor the JM
until the job has finished. Once that has happened, the AM stops Flink on YARN.
To get this `JobID` I needed to make some changes to the CliFrontend /
Client. The Client has two ways of submitting a Job to Flink: an attached mode
(default) and a detached mode.
The attached mode is returning the `JobExecutionResult`, the detached mode
was returning nothing. I created a new type called `JobSubmissionResult` which
is returned by the detached job submission. It only contains the job id. The
JobExecutionResult is extending the JobSubmissionResult.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
