My 2c: KStreams:
Pros: * Streaming as a library: No need to submit your job to a cluster. Easy to scale up/down the job by adding or removing workers. * Streaming durability: State is durably stored in Kafka topics in a streaming fashion. Durability is amortized across the job's lifetime. * No dependencies other than Kafka: If you are already a Kafka shop, then there are no additional dependencies. Cons: * Intra-job traffic goes flows through Kafka: This increases the workload of your Kafka traffic. * State is stored in Kafka: This increases the workload of your Kafka cluster, particularly as they are compacted topics. * Tight Kafka coupling: If you are not a Kafka shop or some other message broker would serve you better. * Parallelism is limited by the number of topic partitions: This is a side effect of the last two issues. Flink: Pros: * Intra-job traffic flows directly between workers. * More mature. * Higher-level constructs: SQL, CEP, etc. Cons: * Requires a cluster to submit a job to: You can't just have some jar and run it. You need either a stand alone or YARN cluster to submit the job to. This makes initial deployment and job deployment complicated. There is some work to alleviate this (e.g. embedding the job with Flink in a container), but its still nowhere as easy as KStreams. * Checkpointed durability: State is durability stored in a distributed filesystem in a checkpoint fashion. If you have a lot of state, checkpointing will intermittently drop the performance of the job and create very spiky network traffic. On Mon, May 20, 2019 at 3:30 PM Peter Groesbeck <peter.groesb...@gmail.com> wrote: > Hi folks, > > I'm hoping to get some deeper clarification on which framework, Flink or > KStreams, to use in a given scenario. I've read over the following blog > article which I think sets a great baseline understanding of the > differences between those frameworks but I would like to get some outside > opinions: > > https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/ > > My understanding of this article is that KStreams works well as an > embedded library in a microservice, API layer, or as a standalone > application at a company with centralized deployment infrastructure, such > as a shared Kubernetes cluster. > > In this case, there is discussion around deploying KStreams as a > standalone application stack backed by EC2 or ECS, and whether or not Flink > is better suited to serve as the data transformation layer. We already do > run Flink applications on EMR. > > The point against Flink is that it requires a cluster whereas KStreams > does not, and can be deployed on ECS or EC2. We do not have a centralized > deployment team, and will still have to maintain either the CNF > Stack/AutoScaling Group or EMR Cluster ourselves. > > What are some of the advantages of using Flink over KStreams standalone? > > The Job management UI is one that comes to mind, and another are some of > the more advanced API options such as CEP. But I would really love to hear > the opinions of people who are familiar with both. In what scenarios would > you choose one over the other? Is it advisable or even preferable to > attempt to deploy KStreams as it's own stack and avoid the complexity of > maintaining a cluster? > > Thanks, > Peter >
