Hi Ruibin, Given that you are starting fresh I would recommend going with Kubernetes and specifically checking out the Flink Kubernetes Operator. [1] I have worked with Yarn for years before I transitioned to Kubernetes a year ago and I am pleased that we made the jump. To address you point on a very high level:
1. Stability: By now both Yarn and Kubernetes are quite stable. 2. Popularity: My educated guess is that Kubernetes is the more popular choice by now. 3. Community support: Kubernetes wins here imho, both in terms of the provider itself and in terms of the Flink integration. See my recent, admittedly biased tweet thread. [2] 4. Toolchains: Hands down Kubernetes. One slight note is that traditionally the Kubernetes toolchain is in Golang, which raises the barrier of entry for Java/Scala folks. By now the Java toolchain for Kubernetes has grown quite a bit though, we are using tools like fabric8 [3] and the Java Operator SDK [4] extensively. Honourable mention is that given the declarative REST APIs of Kubernetes debugging applications is significantly easier than on Yarn. [1] https://github.com/apache/flink-kubernetes-operator [2] https://mobile.twitter.com/MartonBalassi/status/1603498076882214925 [3] https://github.com/fabric8io/kubernetes-client [4] https://javaoperatorsdk.io/ On Mon, Dec 19, 2022 at 7:56 AM Ruibin Xing <xingro...@gmail.com> wrote: > Hi all, > > We are currently setting up a new Flink cluster and are trying to decide > on the best deployment method. As far as we know, Flink supports two > resource providers: YARN and Kubernetes. We are having difficulty > evaluating the pros and cons of each provider, particularly in terms of > stability, popularity, community support, and toolchains. > > Could someone please share their experience with these resource providers > and provide a recommendation for our new cluster? > > Best, Ruibin > >
