[
https://issues.apache.org/jira/browse/FLINK-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078120#comment-16078120
]
ASF GitHub Bot commented on FLINK-7080:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4284
[FLINK-7080] [yarn] Add Yarn per job mode deployment
This PR is based on #4280, #4270, #4271, #4259, #4260, #4261, #4272 and
#4281.
This PR adds the functionality to deploy a Yarn per-job cluster via the
`YarnClusterDescriptorV2.deployJob(ClusterSpecification, JobGraph)`. The
`JobGraph` has to contain the user code jar which is required to run the job.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink addPerJobYarnDeployment
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4284.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4284
----
commit b415fdba6334aa8eeed9fecb3834dbce5ceecc1e
Author: Till Rohrmann <[email protected]>
Date: 2017-07-07T11:40:29Z
[FLINK-7125] [yarn] Remove Configuration loading from
AbstractYarnClusterDescriptor
Instead the AbstractYarnClusterDescriptor is passed in a Configuration
instance which
is sent to the started application master.
Pass in configuration directory manually
Remove configurationDirectory resolution from AbstractYarnClusterDescriptor
commit 8de588bf3b2bc984ecc25603e474df443f5cc265
Author: Till Rohrmann <[email protected]>
Date: 2017-07-06T07:33:18Z
[FLINK-7110] [client] Add per-job cluster deployment interface
commit c68942eb7e154e78bef16930fa8f421f0123483e
Author: Till Rohrmann <[email protected]>
Date: 2017-07-06T12:00:21Z
[FLINK-7113] Make ClusterDescriptor independent of cluster size
The deploySession method now is given a ClusterSpecification which
specifies the
size of the cluster which it is supposed to deploy.
commit 7003cae3c7530b6560dd8ebd365b1b1313a109e0
Author: Till Rohrmann <[email protected]>
Date: 2017-07-04T21:15:52Z
[FLINK-7103] [dispatcher] Add skeletal structure of Dispatcher component
The Dispatcher is responsible for receiving job submissions, persisting the
JobGraphs,
spawning JobManager to execute the jobs and recovering the jobs in case of
a master
failure. This commit adds the basic skeleton including the RPC call for job
submission.
Add cleanup logic for finished jobs
Pass BlobService to JobManagerRunner
commit 24fa27535d7064cd2018c67969f9d2c2ef0238f3
Author: Till Rohrmann <[email protected]>
Date: 2017-07-05T11:38:35Z
[FLINK-7105] Make ActorSystems non daemonic
In order to not having to explicitly wait on the termination of an
ActorSystem
in the main thread, we now create the ActorSystems in non-daemonic mode.
That
way the process won't terminate if there is still an active actor.
commit 60294e77b5d9cb91b9e31c0a80d3514d32c80c88
Author: Till Rohrmann <[email protected]>
Date: 2017-07-05T15:16:52Z
[FLINK-7108] [yarn] Add YARN entry points based on the generic entry point
Add the YarnSesssionClusterEntrypoint and the YarnJobClusterEntrypoint
which extend
SessionClusterEntrypoint and JobClusterEntrypoint, respectively.
Add new Yarn session and per-job cluster entry points
Remove old Flip-6 Yarn per job entry point
commit dee79e55de3575cacdc9ce0d95a4c43974c2d0f6
Author: Till Rohrmann <[email protected]>
Date: 2017-07-06T15:16:18Z
[FLINK-7082] Add generic entry point for session and per-job clusters
commit a940663e7e2894dc1ddd365ca6ae539b52a85c65
Author: Till Rohrmann <[email protected]>
Date: 2017-07-06T15:16:54Z
[FLINK-7086] Add Flip-6 standalone session cluster entry point
commit fce951241e3a4723a6c32b6c08b1aea7513ec397
Author: Till Rohrmann <[email protected]>
Date: 2017-07-06T16:26:41Z
[FLINK-7080] [yarn] Add Yarn per job mode deployment
Upload user code jar from JobGraph
----
> Deploy Yarn cluster with job
> ----------------------------
>
> Key: FLINK-7080
> URL: https://issues.apache.org/jira/browse/FLINK-7080
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management, YARN
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Labels: flip-6
>
> In order to start a yarn per-job cluster, we have to start a Yarn application
> running Flink which includes the job to be executed. One way to do it is to
> upload the serialized form of the {{JobGraph}} as a file. The Yarn entry
> point can then read the {{JobGraph}} from this file and start the
> {{JobManager}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
