yanghua commented on a change in pull request #10777: [FLINK-10939][doc] Add documents for running Flink cluster natively on Kubernetes URL: https://github.com/apache/flink/pull/10777#discussion_r364172284
########## File path: docs/ops/deployment/native_kubernetes.md ########## @@ -0,0 +1,199 @@ +--- +title: "Native Kubernetes Setup" +nav-title: Native Kubernetes +nav-parent_id: deployment +is_beta: true +nav-pos: 7 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +This page describes how to deploy a Flink session cluster natively on [Kubernetes](https://kubernetes.io). + +* This will be replaced by the TOC +{:toc} + +<div class="alert alert-warning"> +Flink's native Kubernetes integration is still experimental. There may be changes in the configuration and CLI flags in latter versions. Job clusters are not yet supported. +</div> + +## Requirements + +- Kubernetes 1.9 or above. +- KubeConfig, which has access to list, create, delete pods and services, configurable via `~/.kube/config`. You can verify permissions by running `kubectl auth can-i <list|create|edit|delete> pods`. +- Kubernetes DNS enabled. +- A service Account with [RBAC](#rbac) permissions to create, delete pods. + +## Flink Kubernetes Session + +### Start Flink Session + +Follow these instructions to start a Flink Session within your Kubernetes cluster. + +A session will start all required Flink services (JobManager and TaskManagers) so that you can submit programs to the cluster. +Note that you can run multiple programs per session. + +{% highlight bash %} +$ ./bin/kubernetes-session.sh +{% endhighlight %} + +All the Kubernetes configuration options can be found in our [configuration guide]({{ site.baseurl }}/ops/config.html#kubernetes). + +**Example**: Issue the following command to start a session cluster with 4 GB of memory and 2 CPUs with 4 slots per TaskManager: + +{% highlight bash %} +./bin/kubernetes-session.sh \ + -Dkubernetes.cluster-id=<ClusterId> \ + -Dtaskmanager.memory.process.size=4096m \ + -Dkubernetes.taskmanager.cpu=2 \ + -Dtaskmanager.numberOfTaskSlots=4 +{% endhighlight %} + +The system will use the configuration in `conf/flink-conf.yaml`. +Please follow our [configuration guide]({{ site.baseurl }}/ops/config.html) if you want to change something. + +If you do not specify a particular name for your session by `kubernetes.cluster-id`, the Flink client will generate a UUID name. + +### Submitting jobs to an existing Session + +Use the following command to submit a Flink Job to the Kubernetes cluster. + +{% highlight bash %} +$ ./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id=<ClusterId> examples/streaming/WindowJoin.jar +{% endhighlight %} + +### Accessing Job Manager UI + +There are several ways to expose a Service onto an external (outside of your cluster) IP address. +This can be configured using `kubernetes.service.exposed.type`. + +- `ClusterIP`: Exposes the service on a cluster-internal IP. +The Service is only reachable within the cluster. If you want to access the Job Manager ui or submit job to the existing session, you need to start a local proxy. +You can then use `localhost:8081` to submit a Flink job to the session or view the dashboard. + +{% highlight bash %} +$ kubectl port-forward service/<ServiceName> 8081 +{% endhighlight %} + +- `NodePort`: Exposes the service on each Node’s IP at a static port (the `NodePort`). `<NodeIP>:<NodePort>` could be used to contact the Job Manager Service. `NodeIP` could be easily replaced with Kubernetes ApiServer address. +You could find it in your kube config file. + +- `LoadBalancer`: Default value, exposes the service externally using a cloud provider’s load balancer. +Since the cloud provider and Kubernetes needs some time to prepare the load balancer, you may get a `NodePort` JobManager Web Interface in the client log. +You can use `kubectl get services/<ClusterId>` to get EXTERNAL-IP and then construct the load balancer JobManager Web Interface manually `http://<EXTERNAL-IP>:8081`. + +- `ExternalName`: Map a service to a DNS name, not supported in current version. + +Please reference the official documentation on [publishing services in Kubernetes](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) for more information. + +### Attach to an existing Session + +The Kubernetes session is started in detached mode by default, meaning the Flink client will exit after submitting all the resources to the Kubernetes cluster. Use the following command to attach to an existing session. + +{% highlight bash %} +$ ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true +{% endhighlight %} + +### Stop Flink Session + +To stop a Flink Kubernetes session, attach the Flink client to the cluster and type `stop`. + +{% highlight bash %} +$ echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterId> -Dexecution.attached=true +{% endhighlight %} + +#### Manual Resource Cleanup + +Flink uses [Kubernetes ownerReference's](https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/) to cleanup all cluster components. +All the Flink created resources, including `ConfigMap`, `Service`, `Deployment`, `Pod`, have been set the ownerReference to `service/<ClusterId>`. +When the service is deleted, all other resource will be deleted automatically. Review comment: resource -> resources ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
