morsapaes commented on a change in pull request #417: URL: https://github.com/apache/flink-web/pull/417#discussion_r573115924
########## File path: _posts/2021-02-10-native-k8s-with-ha.md ########## @@ -0,0 +1,112 @@ +--- +layout: post +title: "How to natively deploy Flink on Kubernetes with HA" +date: 2021-02-10 00:00:00 +authors: +- Yang Wang: + name: "Yang Wang" +excerpt: Kubernetes provides built-in functionalities that Flink can leverage for JobManager failover. In Flink 1.12 (FLIP-144), the community implemented a Kubernetes HA service as an alternative to ZooKeeper for highly available production setups. In this blogpost, we will have a close look at how to deploy Flink applications natively on Kubernetes cluster with HA. +--- + +Flink has supported resource management systems like YARN and Mesos since the early days; however, these were not designed for the fast-moving cloud-native architectures that are increasingly gaining popularity these days, or the growing need to support complex, mixed workloads (e.g. batch, streaming, deep learning, web services). +For these reasons, more and more users are turning to Kubernetes to automate the deployment, scaling and management of their Flink applications. + +From release to release, the Flink community has made significant progress in integrating natively with Kubernetes, from active resource management to “Zookeeperless” High Availability (HA). +In this blogpost, we will go through the technical details of deploying Flink applications natively on Kubernetes. +Then, we’ll deep dive into Flink’s Kubernetes High Availability (HA) architecture and walk through a hands-on example of running a Flink [application cluster](https://ci.apache.org/projects/flink/flink-docs-stable/concepts/glossary.html#flink-application-cluster) on Kubernetes with HA enabled. +We’ll end with a conclusion covering the advantages of running Flink on Kubernetes, and an outlook into future work. + +# Native Flink on Kubernetes Architecture + +Before we dive into the technical details of how the native Kubernetes integration works, let us figure out what “native” means here: + + 1. Flink is self-contained. There will be an embedded Kubernetes client in the Flink client, and so you will not need other external tools (e.g. kubectl, Kubernetes dashboard) to create a Flink cluster on Kubernetes. + 2. The Flink client will contact the Kubernetes API server directly to create the JobManager deployment. The configuration located on the client side will be shipped to the JobManager pod, as well as the log4j and hadoop configuration. + 3. Flink’s ResourceManager will talk to the Kubernetes API server to allocate and release the TaskManager pods dynamically on-demand. + +All in all, this is similar to how Flink integrates with other resource management systems (e.g. YARN, Mesos), so it should be somewhat straightforward to integrate with Kubernetes if you’ve managed such deployments — especially so if you already have some internal deployer for the lifecycle management of multiple Flink jobs. + +<center> +<img vspace="8" style="width:75%" src="{{site.baseurl}}/img/blog/2021-02-10-native-k8s-with-ha/native-k8s-architecture.png" /> +</center> + +# Kubernetes High Availability Service + +High Availability (HA) is a common requirement when bringing Flink to production: it helps prevent a single point of failure for Flink clusters. Review comment: Mostly a matter of preference here. Both forms are correct, adding the "to" just makes it a tad bit more formal. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
