This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch release-1.6
in repository https://gitbox.apache.org/repos/asf/flink.git
commit a12d087a17d64f748ede472b369cea44c56c923f
Author: Till Rohrmann <trohrm...@apache.org>
AuthorDate: Tue Aug 14 14:39:14 2018 +0200
[FLINK-10001][docs] Add documentation for job cluster deployment on Docker
and K8s
[FLINK-10001][docs] Add documentation for job cluster deployment on K8s
This closes #6562.
---
docs/ops/deployment/docker.md | 42 +++++++++----
docs/ops/deployment/kubernetes.md | 67 ++++++++++++++------
flink-container/docker/README.md | 72 ++++++++++++++++------
flink-container/docker/docker-compose.yml | 13 +++-
flink-container/kubernetes/README.md | 11 ++--
.../container-scripts/docker-compose.test.yml | 2 +-
6 files changed, 149 insertions(+), 58 deletions(-)
diff --git a/docs/ops/deployment/docker.md b/docs/ops/deployment/docker.md
index 4986f2a..453693d 100644
--- a/docs/ops/deployment/docker.md
+++ b/docs/ops/deployment/docker.md
@@ -23,20 +23,24 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime. There are
-official Docker images for Apache Flink available on Docker Hub which can be
-used directly or extended to better integrate into a production environment.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available on Docker Hub which can be
used to deploy a session cluster.
+The Flink repository also contains tooling to create container images to
deploy a job cluster.
* This will be replaced by the TOC
{:toc}
-## Official Docker Images
+## Flink session cluster
+
+A Flink session cluster can be used to run multiple jobs.
+Each job needs to be submitted to the cluster after it has been deployed.
+
+### Docker images
The [official Docker repository](https://hub.docker.com/_/flink/) is
hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
-Images for each supported combination of Hadoop and Scala are available, and
-tag aliases are provided for convenience.
+Images for each supported combination of Hadoop and Scala are available, and
tag aliases are provided for convenience.
For example, the following aliases can be used: *(`1.2.y` indicates the latest
release of Flink 1.2)*
@@ -63,13 +67,25 @@ For example:
**Note:** The docker images are provided as a community project by individuals
on a best-effort basis. They are not official releases by the Apache Flink PMC.
+## Flink job cluster
+
+A Flink job cluster is a dedicated cluster which runs a single job.
+The job is part of the image and, thus, there is no extra job submission
needed.
+
+### Docker images
+
+The Flink job cluster image needs to contain the user code jars of the job for
which the cluster is started.
+Therefore, one needs to build a dedicated container image for every job.
+The `flink-container` module contains a `build.sh` script which can be used to
create such an image.
+Please see the [instructions](https://github.com/apache/flink/blob/{{
site.github_branch }}/flink-container/docker/README.md) for more details.
+
## Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a convenient way to run a
group of Docker containers locally.
-An [example config
file](https://github.com/docker-flink/examples/blob/master/docker-compose.yml)
-is available on GitHub.
+Example config files for a [session
cluster](https://github.com/docker-flink/examples/blob/master/docker-compose.yml)
and a [job cluster](https://github.com/apache/flink/blob/{{ site.github_branch
}}/flink-container/docker/docker-compose.yml)
+are available on GitHub.
### Usage
@@ -85,10 +101,14 @@ is available on GitHub.
docker-compose scale taskmanager=<N>
-When the cluster is running, you can visit the web UI at [http://localhost:8081
-](http://localhost:8081) and submit a job.
+* Kill the cluster
+
+ docker-compose kill
+
+When the cluster is running, you can visit the web UI at
[http://localhost:8081](http://localhost:8081).
+You can also use the web UI to submit a job to a session cluster.
-To submit a job via the command line, you must copy the JAR to the Jobmanager
+To submit a job to a session cluster via the command line, you must copy the
JAR to the JobManager
container and submit the job from there.
For example:
diff --git a/docs/ops/deployment/kubernetes.md
b/docs/ops/deployment/kubernetes.md
index 37489fe..298e473 100644
--- a/docs/ops/deployment/kubernetes.md
+++ b/docs/ops/deployment/kubernetes.md
@@ -23,51 +23,78 @@ specific language governing permissions and limitations
under the License.
-->
-[Kubernetes](https://kubernetes.io) is a container orchestration system.
+This page describes how to deploy a Flink job and session cluster on
[Kubernetes](https://kubernetes.io).
* This will be replaced by the TOC
{:toc}
-## Simple Kubernetes Flink Cluster
+## Setup Kubernetes
-A basic Flink cluster deployment in Kubernetes has three components:
+Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in
order to deploy a Kubernetes cluster.
+If you want to run Kubernetes locally, we recommend using
[MiniKube](https://kubernetes.io/docs/setup/minikube/).
-* a Deployment for a single Jobmanager
-* a Deployment for a pool of Taskmanagers
-* a Service exposing the Jobmanager's RPC and UI ports
+<div class="alert alert-info" markdown="span">
+ <strong>Note:</strong> If using MiniKube please make sure to execute
`minikube ssh 'sudo ip link set docker0
+ promisc on'` before deploying a Flink cluster. Otherwise Flink components
are not able to self reference
+ themselves through a Kubernetes service.
+</div>
-### Launching the cluster
+## Flink session cluster on Kubernetes
-Using the [resource definitions found below](#simple-kubernetes-flink-cluster-
-resources), launch the cluster with the `kubectl` command:
+A Flink session cluster is executed as a long-running Kubernetes Deployment.
+Note that you can run multiple Flink jobs on a session cluster.
+Each job needs to be submitted to the cluster after the cluster has been
deployed.
+
+A basic Flink session cluster deployment in Kubernetes has three components:
+
+* a Deployment/Job which runs the JobManager
+* a Deployment for a pool of TaskManagers
+* a Service exposing the JobManager's REST and UI ports
+
+### Deploy Flink session cluster on Kubernetes
+
+Using the resource definitions for a [session
cluster](#session-cluster-resource-definitions), launch the cluster with the
`kubectl` command:
- kubectl create -f jobmanager-deployment.yaml
kubectl create -f jobmanager-service.yaml
+ kubectl create -f jobmanager-deployment.yaml
kubectl create -f taskmanager-deployment.yaml
You can then access the Flink UI via `kubectl proxy`:
1. Run `kubectl proxy` in a terminal
-2. Navigate to
[http://localhost:8001/api/v1/proxy/namespaces/default/services/flink-jobmanager:8081
-](http://localhost:8001/api/v1/proxy/namespaces/default/services/flink-
-jobmanager:8081) in your browser
+2. Navigate to
[http://localhost:8001/api/v1/namespaces/default/services/flink-jobmanager:ui/proxy](http://localhost:8001/api/v1/namespaces/default/services/flink-jobmanager:ui/proxy)
in your browser
-### Deleting the cluster
-
-Again, use `kubectl` to delete the cluster:
+In order to terminate the Flink session cluster, use `kubectl`:
kubectl delete -f jobmanager-deployment.yaml
- kubectl delete -f jobmanager-service.yaml
kubectl delete -f taskmanager-deployment.yaml
+ kubectl delete -f jobmanager-service.yaml
+
+## Flink job cluster on Kubernetes
+
+A Flink job cluster is a dedicated cluster which runs a single job.
+The job is part of the image and, thus, there is no extra job submission
needed.
+
+### Creating the job-specific image
+
+The Flink job cluster image needs to contain the user code jars of the job for
which the cluster is started.
+Therefore, one needs to build a dedicated container image for every job.
+Please follow these [instructions](https://github.com/apache/flink/blob/{{
site.github_branch }}/flink-container/docker/README.md) to build the Docker
image.
+
+### Deploy Flink job cluster on Kubernetes
+
+In order to deploy the a job cluster on Kubernetes please follow these
[instructions](https://github.com/apache/flink/blob/{{ site.github_branch
}}/flink-container/kubernetes/README.md#deploy-flink-job-cluster).
## Advanced Cluster Deployment
-An early version of a [Flink Helm chart](https://github.com/docker-flink/
-examples) is available on GitHub.
+An early version of a [Flink Helm
chart](https://github.com/docker-flink/examples) is available on GitHub.
## Appendix
-### Simple Kubernetes Flink cluster resources
+### Session cluster resource definitions
+
+The Deployment definitions use the pre-built image `flink:latest` which can be
found [on Docker Hub](https://hub.docker.com/r/_/flink/).
+The image is built from this [Github
repository](https://github.com/docker-flink/docker-flink).
`jobmanager-deployment.yaml`
{% highlight yaml %}
diff --git a/flink-container/docker/README.md b/flink-container/docker/README.md
index 644b31c..7d3c030 100644
--- a/flink-container/docker/README.md
+++ b/flink-container/docker/README.md
@@ -1,40 +1,74 @@
-# Apache Flink job cluster deployment on docker using docker-compose
+# Apache Flink job cluster Docker image
-## Installation
+In order to deploy a job cluster on Docker, one needs to create an image which
contains the Flink binaries as well as the user code jars of the job to execute.
+This directory contains a `build.sh` which facilitates the process.
+The script takes a Flink distribution either from an official release, an
archive or a local distribution and combines it with the specified job jar.
-Install the most recent stable version of docker
-https://docs.docker.com/installation/
+## Installing Docker
-## Build
+Install the most recent stable version of
[Docker](https://docs.docker.com/installation/).
-Images are based on the official Java Alpine (OpenJDK 8) image. If you want to
-build the flink image run:
+## Building the Docker image
- build.sh --from-local-dist --job-jar /path/to/job/jar/job.jar --image-name
flink:job
+Images are based on the official Java Alpine (OpenJDK 8) image.
-If you want to build the container for a specific version of flink/hadoop/scala
-you can configure it in the respective args:
+Before building the image, one needs to build the user code jars for the job.
+Assume that the job jar is stored under `<PATH_TO_JOB_JAR>`
- docker build --build-arg FLINK_VERSION=1.6.0 --build-arg HADOOP_VERSION=28
--build-arg SCALA_VERSION=2.11 -t "flink:1.6.0-hadoop2.8-scala_2.11" flink
+If you want to build the Flink image from the version you have checked out
locally run:
-## Deploy
+ build.sh --from-local-dist --job-jar <PATH_TO_JOB_JAR> --image-name
<IMAGE_NAME>
+
+Note that you first need to call `mvn package -pl flink-dist -am` to build the
Flink binaries.
-- Deploy cluster and see config/setup log output (best run in a screen session)
+If you want to build the Flink image from an archive stored under
`<PATH_TO_ARCHIVE>` run:
- docker-compose up
+ build.sh --from-archive <PATH_TO_ARCHIVE> --job-jar <PATH_TO_JOB_JAR>
--image-name <IMAGE_NAME>
-- Deploy as a daemon (and return)
+If you want to build the Flink image for a specific version of
Flink/Hadoop/Scala run:
- docker-compose up -d
+ build.sh --from-release --flink-version 1.6.0 --hadoop-version 2.8
--scala-version 2.11 --image-name <IMAGE_NAME>
+
+The script will try to download the released version from the Apache archive.
-- Scale the cluster up or down to *N* TaskManagers
+## Deploying via Docker compose
+
+The `docker-compose.yml` contains the following parameters:
+
+* `FLINK_DOCKER_IMAGE_NAME` - Image name to use for the deployment (default:
`flink-job:latest`)
+* `FLINK_JOB` - Name of the Flink job to execute (default: none)
+* `DEFAULT_PARALLELISM` - Default parallelism with which to start the job
(default: 1)
+* `FLINK_JOB_ARGUMENTS` - Additional arguments which will be passed to the job
cluster (default: none)
+
+The parameters can be set by exporting the corresponding environment variable.
+
+Deploy cluster and see config/setup log output (best run in a screen session)
+
+ FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME>
docker-compose up
+
+Deploy as a daemon (and return)
+
+ FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME>
docker-compose up -d
+
+In order to start the job with a different default parallelism set
`DEFAULT_PARALLELISM`.
+This will automatically start `DEFAULT_PARALLELISM` TaskManagers:
+
+ FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME>
DEFAULT_PARALLELISM=<DEFAULT_PARALLELISM> docker-compose up
+
+One can also provide additional job arguments via `FLINK_JOB_ARGUMENTS` which
are passed to the job:
+
+ FLINK_DOCKER_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME>
FLINK_JOB_ARGUMENTS=<JOB_ARGUMENTS> docker-compose up
+
+Scale the cluster up or down to *N* TaskManagers
docker-compose scale taskmanager=<N>
-- Access the Job Manager container
+Access the Job Manager container
docker exec -it $(docker ps --filter name=flink_jobmanager
--format={{.ID}}) /bin/sh
+
+Access the web UI by going to `<IP_DOCKER_MACHINE>:8081` in your web browser.
-- Kill the cluster
+Kill the cluster
docker-compose kill
diff --git a/flink-container/docker/docker-compose.yml
b/flink-container/docker/docker-compose.yml
index 5fddff3..28b5368 100644
--- a/flink-container/docker/docker-compose.yml
+++ b/flink-container/docker/docker-compose.yml
@@ -16,16 +16,23 @@
# limitations under the License.
################################################################################
-# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image
name to use
+# Docker compose file for a Flink job cluster deployment.
+#
+# Parameters:
+# * FLINK_DOCKER_IMAGE_NAME - Image name to use for the deployment (default:
flink-job:latest)
+# * FLINK_JOB - Name of the Flink job to execute (default: none)
+# * DEFAULT_PARALLELISM - Default parallelism with which to start the job
(default: 1)
+# * FLINK_JOB_ARGUMENTS - Additional arguments which will be passed to the job
cluster (default: none)
-version: "2.1"
+version: "2.2"
services:
job-cluster:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
ports:
- "8081:8081"
- command: job-cluster --job-classname ${FLINK_JOB}
-Djobmanager.rpc.address=job-cluster ${FLINK_JOB_ARGUMENTS}
+ command: job-cluster --job-classname ${FLINK_JOB}
-Djobmanager.rpc.address=job-cluster
-Dparallelism.default=${DEFAULT_PARALLELISM:-1} ${FLINK_JOB_ARGUMENTS}
taskmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
command: task-manager -Djobmanager.rpc.address=job-cluster
+ scale: ${DEFAULT_PARALLELISM:-1}
diff --git a/flink-container/kubernetes/README.md
b/flink-container/kubernetes/README.md
index 5fd2286..efba823 100644
--- a/flink-container/kubernetes/README.md
+++ b/flink-container/kubernetes/README.md
@@ -22,26 +22,29 @@ The files contain the following variables:
One way to substitute the variables is to use `envsubst`.
See [here](https://stackoverflow.com/a/23622446/4815083) for a guide to
install it on Mac OS X.
+Alternatively, copy the template files (suffixed with `*.template`) and
replace the variables.
+
In non HA mode, you should first start the job cluster service:
`kubectl create -f job-cluster-service.yaml`
In order to deploy the job cluster entrypoint run:
-`FLINK_IMAGE_NAME=<job-image> FLINK_JOB=<job-name>
FLINK_JOB_PARALLELISM=<parallelism> envsubst < job-cluster-job.yaml.template |
kubectl create -f -`
+`FLINK_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB=<JOB_NAME>
FLINK_JOB_PARALLELISM=<PARALLELISM> envsubst < job-cluster-job.yaml.template |
kubectl create -f -`
Now you should see the `flink-job-cluster` job being started by calling
`kubectl get job`.
At last, you should start the task manager deployment:
-`FLINK_IMAGE_NAME=<job-image> FLINK_JOB_PARALLELISM=<parallelism> envsubst <
task-manager-deployment.yaml.template | kubectl create -f -`
+`FLINK_IMAGE_NAME=<IMAGE_NAME> FLINK_JOB_PARALLELISM=<PARALLELISM> envsubst <
task-manager-deployment.yaml.template | kubectl create -f -`
## Interact with Flink job cluster
-After starting the job cluster service, the web UI will be available under
`<NodeIP>:30081`.
+After starting the job cluster service, the web UI will be available under
`<NODE_IP>:30081`.
+In the case of Minikube, `<NODE_IP>` equals `minikube ip`.
You can then use the Flink client to send Flink commands to the cluster:
-`bin/flink list -m <NodeIP:30081>`
+`bin/flink list -m <NODE_IP:30081>`
## Terminate Flink job cluster
diff --git
a/flink-end-to-end-tests/test-scripts/container-scripts/docker-compose.test.yml
b/flink-end-to-end-tests/test-scripts/container-scripts/docker-compose.test.yml
index 5281111..7af9183 100644
---
a/flink-end-to-end-tests/test-scripts/container-scripts/docker-compose.test.yml
+++
b/flink-end-to-end-tests/test-scripts/container-scripts/docker-compose.test.yml
@@ -18,7 +18,7 @@
# Extensions to flink-container/docker/docker-compose.yml that mounts volumes
needed for tests
-version: "2.1"
+version: "2.2"
services:
job-cluster:
volumes:
