Repository: flink Updated Branches: refs/heads/master 3642c5a60 -> 71d76731d
[FLINK-6330] [docs] Add basic Docker, K8s docs This closes #3751 Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/91f37658 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/91f37658 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/91f37658 Branch: refs/heads/master Commit: 91f376589b717d46b124d7f8e181950926f2ca1e Parents: 3642c5a Author: Patrick Lucas <[email protected]> Authored: Fri Apr 21 15:00:53 2017 +0200 Committer: Greg Hogan <[email protected]> Committed: Wed May 10 13:27:21 2017 -0400 ---------------------------------------------------------------------- docs/docker/run.sh | 4 +- docs/setup/docker.md | 99 ++++++++++++++++++++++++++ docs/setup/kubernetes.md | 157 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 259 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/91f37658/docs/docker/run.sh ---------------------------------------------------------------------- diff --git a/docs/docker/run.sh b/docs/docker/run.sh index 3c8878a..5598c0a 100755 --- a/docs/docker/run.sh +++ b/docs/docker/run.sh @@ -31,10 +31,12 @@ if [ "$(uname -s)" == "Linux" ]; then USER_NAME=${SUDO_USER:=$USER} USER_ID=$(id -u "${USER_NAME}") GROUP_ID=$(id -g "${USER_NAME}") + LOCAL_HOME="/home/${USER_NAME}" else # boot2docker uid and gid USER_NAME=$USER USER_ID=1000 GROUP_ID=50 + LOCAL_HOME="/Users/${USER_NAME}" fi docker build -t "${IMAGE_NAME}-${USER_NAME}" - <<UserSpecificDocker @@ -65,7 +67,7 @@ docker run -i -t \ -w ${FLINK_DOC_ROOT} \ -u "${USER}" \ -v "${FLINK_DOC_ROOT}:${FLINK_DOC_ROOT}" \ - -v "/home/${USER_NAME}:/home/${USER_NAME}" \ + -v "${LOCAL_HOME}:/home/${USER_NAME}" \ -p 4000:4000 \ ${IMAGE_NAME}-${USER_NAME} \ bash -c "${CMD}" http://git-wip-us.apache.org/repos/asf/flink/blob/91f37658/docs/setup/docker.md ---------------------------------------------------------------------- diff --git a/docs/setup/docker.md b/docs/setup/docker.md new file mode 100644 index 0000000..29e696f --- /dev/null +++ b/docs/setup/docker.md @@ -0,0 +1,99 @@ +--- +title: "Docker Setup" +nav-title: Docker +nav-parent_id: deployment +nav-pos: 4 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +[Docker](https://www.docker.com) is a popular container runtime. There are +official Flink Docker images available on Docker Hub which can be used directly +or extended to better integrate into a production environment. + +* This will be replaced by the TOC +{:toc} + +## Official Flink Docker Images + +The [official Flink Docker repository](https://hub.docker.com/_/flink/) is +hosted on Docker Hub and serves images of Flink version 1.2.1 and later. + +Images for each supported combination of Hadoop and Scala are available, and +tag aliases are provided for convenience. + +For example, the following aliases can be used: *(`1.2.y` indicates the latest +release of Flink 1.2)* + +* `flink:latest` â +`flink:<latest-flink>-hadoop<latest-hadoop>-scala_<latest-scala>` +* `flink:1.2` â `flink:1.2.y-hadoop27-scala_2.11` +* `flink:1.2.1-scala_2.10` â `flink:1.2.1-hadoop27-scala_2.10` +* `flink:1.2-hadoop26` â `flink:1.2.y-hadoop26-scala_2.11` + +<!-- NOTE: uncomment when docker-flink/docker-flink/issues/14 is resolved. --> +<!-- +Additionally, images based on Alpine Linux are available. Reference them by +appending `-alpine` to the tag. For the Alpine version of `flink:latest`, use +`flink:alpine`. + +For example: + +* `flink:alpine` +* `flink:1.2.1-alpine` +* `flink:1.2-scala_2.10-alpine` +--> + +## Flink with Docker Compose + +[Docker Compose](https://docs.docker.com/compose/) is a convenient way to run a +group of Docker containers locally. + +An [example config file](https://github.com/docker-flink/examples) is available +on GitHub. + +### Usage + +* Launch a cluster in the foreground + + docker-compose up + +* Launch a cluster in the background + + docker-compose up -d + +* Scale the cluster up or down to *N* TaskManagers + + docker-compose scale taskmanager=<N> + +When the cluster is running, you can visit the web UI at [http://localhost:8081 +](http://localhost:8081) and submit a job. + +To submit a job via the command line, you must copy the JAR to the Jobmanager +container and submit the job from there. + +For example: + +{% raw %} + $ JOBMANAGER_CONTAINER=$(docker ps --filter name=jobmanager --format={{.ID}}) + $ docker cp path/to/jar "$JOBMANAGER_CONTAINER":/job.jar + $ docker exec -t -i "$JOBMANAGER_CONTAINER" flink run /job.jar +{% endraw %} + +{% top %} http://git-wip-us.apache.org/repos/asf/flink/blob/91f37658/docs/setup/kubernetes.md ---------------------------------------------------------------------- diff --git a/docs/setup/kubernetes.md b/docs/setup/kubernetes.md new file mode 100644 index 0000000..0790a05 --- /dev/null +++ b/docs/setup/kubernetes.md @@ -0,0 +1,157 @@ +--- +title: "Kubernetes Setup" +nav-title: Kubernetes +nav-parent_id: deployment +nav-pos: 4 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +[Kubernetes](https://kubernetes.io) is a container orchestration system. + +* This will be replaced by the TOC +{:toc} + +## Simple Kubernetes Flink Cluster + +A basic Flink cluster deployment in Kubernetes has three components: + +* a Deployment for a single Jobmanager +* a Deployment for a pool of Taskmanagers +* a Service exposing the Jobmanager's RPC and UI ports + +### Launching the cluster + +Using the [resource definitions found below](#simple-kubernetes-flink-cluster- +resources), launch the cluster with the `kubectl` command: + + kubectl create -f jobmanager-deployment.yaml + kubectl create -f taskmanager-deployment.yaml + kubectl create -f jobmanager-service.yaml + +You can then access the Flink UI via `kubectl proxy`: + +1. Run `kubectl proxy` in a terminal +2. Navigate to [http://localhost:8001/api/v1/proxy/namespaces/default/services/flink-jobmanager:8081 +](http://localhost:8001/api/v1/proxy/namespaces/default/services/flink- +jobmanager:8081) in your browser + +### Deleting the cluster + +Again, use `kubectl` to delete the cluster: + + kubectl delete -f jobmanager-deployment.yaml + kubectl delete -f taskmanager-deployment.yaml + kubectl delete -f jobmanager-service.yaml + +## Advanced Cluster Deployment + +An early version of a [Flink Helm chart](https://github.com/docker-flink/ +examples) is available on GitHub. + +## Appendix + +### Simple Kubernetes Flink cluster resources + +`jobmanager-deployment.yaml` +{% highlight yaml %} +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: flink-jobmanager +spec: + replicas: 1 + template: + metadata: + labels: + app: flink + component: jobmanager + spec: + containers: + - name: jobmanager + image: flink:latest + args: + - jobmanager + ports: + - containerPort: 6123 + name: rpc + - containerPort: 6124 + name: blob + - containerPort: 6125 + name: query + - containerPort: 8081 + name: ui + env: + - name: JOB_MANAGER_RPC_ADDRESS + value: flink-jobmanager +{% endhighlight %} + +`taskmanager-deployment.yaml` +{% highlight yaml %} +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: flink-taskmanager +spec: + replicas: 2 + template: + metadata: + labels: + app: flink + component: taskmanager + spec: + containers: + - name: taskmanager + image: flink:latest + args: + - taskmanager + ports: + - containerPort: 6121 + name: data + - containerPort: 6122 + name: rpc + - containerPort: 6125 + name: query + env: + - name: JOB_MANAGER_RPC_ADDRESS + value: flink-jobmanager +{% endhighlight %} + +`jobmanager-service.yaml` +{% highlight yaml %} +apiVersion: v1 +kind: Service +metadata: + name: flink-jobmanager +spec: + ports: + - name: rpc + port: 6123 + - name: blob + port: 6124 + - name: query + port: 6125 + - name: ui + port: 8081 + selector: + app: flink + component: jobmanager +{% endhighlight %} + +{% top %}
