Thanks Yang. K8s natively integration is very necessary and important for the adoption of flink IMO. I notice that the design doc is written in 2018, is there any changes or update ?
>>> Download the flink release binary and create the ~/.kube/config file corresponding to the k8s cluster. It is all what you need. How can I specify which k8s cluster to run in case I have multiple k8s clusters ? Can I do it via specifying flink cluster in flink cli ? Yang Wang <danrtsey...@gmail.com> 于2019年8月9日周五 下午9:12写道: > Hi all, > > Currently cloud native architectures has been introduced to many companies > in production. They use kubernetes to run deep learning, web server, etc. > If we could deploy the per-job/session flink cluster on kubernetes to make > it mix-run with other workloads, the cluster resource utilization will be > better. Also many kubernetes users are more easier to have a taste on the > flink. > > By now we have three options to run flink jobs on k8s. > > [1]. Create jm/tm/service yaml and apply, then you will get a flink > standalone cluster on k8s. Use flink run to submit job to the existed flink > cluster. Some companies may have their own deploy system to manage the > flink cluster. > > [2]. Use flink-k8s-operator to manage multiple flink clusters, including > session and perjob. It could manage the complete deployment lifecycle of > the application. I think this option is really easy to use for the k8s > users. They are familiar with k8s-opertor, kubectl and other tools of k8s. > They could debug and run the flink cluster just like other k8s > applications. > > [3]. Natively integration with k8s, use the flink run or > kubernetes-session.sh to start a flink cluster. It is very similar to > submitting an flink cluster to Yarn. KubernetesClusterDescriptor talks to > k8s api server to start a flink master deployment of 1. > KubernetesResourceManager dynamically allocates resource from k8s to start > task manager as demand. This option is very easy for flink users to get > started. In the simplest case, we just need to update the '-m yarn-cluster' > to -m '-m kubernetes-cluster'. > > We have make an internal implementation of option [3] and use it in > production. After fully tested, we hope to contribute it to the community. > Now we want to get some feedbacks about the three options. Any comments are > welcome. > > > > What do we need to prepare when start a flink cluster on k8s using native > integration? > > Download the flink release binary and create the ~/.kube/config file > corresponding to the k8s cluster. It is all what you need. > > > > Flink Session cluster > > * start a session cluster > > ./bin/kubernetes-session.sh -d -n 2 -tm 512 -s 4 -nm flink-session-example > -i flink:latest -kD kubernetes.service.exposed.type=NODE_PORT > > * You will get an address to submit job, specify it through ’-ksa’ option > > ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-session-example > -ksa {x.x.x.x:12345} examples/streaming/WindowJoin.jar > > > > Flink Job Cluster > > * running with official flink image > > ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1 > -ki flink:latest examples/streaming/WindowJoin.jar > > * running with user image > > ./bin/flink run -d -p 4 -m kubernetes-cluster -knm flink-perjob-example-1 > -ki flink-user:latest examples/streaming/WindowJoin.jar > > > > [1]. > > https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html > > [2].https://github.com/lyft/flinkk8soperator > > [3]. > > https://docs.google.com/document/d/1Zmhui_29VASPcBOEqyMWnF3L6WEWZ4kedrCqya0WaAk/edit# > -- Best Regards Jeff Zhang