Re: Flink native k8s integration vs. operator

2022-01-20 Thread Alexis Sarda-Espinosa
rnetes Regards, Alexis. From: Gyula Fóra mailto:gyula.f...@gmail.com>> Sent: Montag, 17. Januar 2022 12:40 To: dev mailto:d...@flink.apache.org>> Cc: Xintong Song mailto:tonysong...@gmail.com>>; user mailto:user@flink.apache.org>> Subject: Re: Flink native k8s integ

Re: Flink native k8s integration vs. operator

2022-01-20 Thread Robert Metzger
ol > containers such as Spring. I only used javax annotations (soon to be > jakarta), so it’s not tied to Spring. > > > > Something I haven’t considered at all in my code is ingress for Flink’s UI. > > > > Let me know what you think. > > > > [1] > https:

Re: Flink native k8s integration vs. operator

2022-01-17 Thread Gyula Fóra
Hi Yang! Thanks for the input! I agree with you on both points that you made. Even if we might support both standalone and native modes in the long run, we should probably build the first version on top of the native integration. This I feel will result in a much simpler, minimalistic first

Re: Flink native k8s integration vs. operator

2022-01-17 Thread Yang Wang
Glad to see that the interest of this thread keeps going. And thanks Thomas, Gyula, and Marton for driving this effort. I want to share my two cents about the Flink K8s operator. > Standalone deployment VS native K8s integration There is already some feature requirement issue[1] for the

Re: Flink native k8s integration vs. operator

2022-01-13 Thread Xintong Song
Thanks for volunteering to drive this effort, Marton, Thomas and Gyula. Looking forward to the public discussion. Please feel free to reach out if there's anything you need from us. Thank you~ Xintong Song On Fri, Jan 14, 2022 at 8:27 AM Chenya Zhang wrote: > Thanks Thomas, Gyula, and

Re: Flink native k8s integration vs. operator

2022-01-13 Thread Chenya Zhang
Thanks Thomas, Gyula, and Marton for driving this effort! It would greatly ease the adoption of Apache Flink on Kubernetes and help to address the current operational pain points as mentioned. Look forward to the proposal and more discussions! Best, Chenya On Thu, Jan 13, 2022 at 12:15 PM Márton

Re: Flink native k8s integration vs. operator

2022-01-13 Thread Márton Balassi
Hi All, I am pleased to see the level of enthusiasm and technical consideration already emerging in this thread. I wholeheartedly support building an operator and endorsing it via placing it under the Apache Flink umbrella (as a separate repository) as the current lack of it is clearly becoming

Re: Flink native k8s integration vs. operator

2022-01-13 Thread Konstantin Knauf
Hi Thomas, Yes, I was referring to a separate repository under Apache Flink. Cheers, Konstantin On Thu, Jan 13, 2022 at 6:19 AM Thomas Weise wrote: > Hi everyone, > > Thanks for the feedback and discussion. A few additional thoughts: > > [Konstantin] > With respect to common lifecycle

Re: Flink native k8s integration vs. operator

2022-01-12 Thread Thomas Weise
Hi everyone, Thanks for the feedback and discussion. A few additional thoughts: [Konstantin] > With respect to common lifecycle management operations: these features are > not available (within Apache Flink) for any of the other resource providers > (YARN, Standalone) either. From this

Re: Flink native k8s integration vs. operator

2022-01-12 Thread Konstantin Knauf
cc dev@ Hi Thomas, Hi everyone, Thank you for starting this discussion and sorry for chiming in late. I agree with Thomas' and David's assessment of Flink's "Native Kubernetes Integration", in particular, it does actually not integrate well with the Kubernetes ecosystem despite being called

Re: Flink native k8s integration vs. operator

2022-01-10 Thread Gyula Fóra
Hi All! This is a very interesting discussion. I think many users find it confusing what deployment mode to choose when considering a new production application on Kubernetes. With all the options of native, standalone and different operators this can get tricky :) I really like the idea that

Re: Flink native k8s integration vs. operator

2022-01-09 Thread Yang Wang
Thanks all for this fruitful discussion. I think Xintong has given a strong point why we introduced the native K8s integration, which is active resource management. I have a concrete example for this in the production. When a K8s node is down, the standalone K8s deployment will take longer

Re: Flink native k8s integration vs. operator

2022-01-06 Thread Xintong Song
Hi folks, Thanks for the discussion. I'd like to share my two cents on this topic. Firstly, I'd like to clarify my understanding of the concepts "native k8s integration" and "active resource management". - Native k8s integration means Flink's master interacts with k8s' api server directly. It

Re: Flink native k8s integration vs. operator

2022-01-05 Thread Thomas Weise
Hi David, Thank you for the reply and context! As for workload types and where native integration might fit: I think that any k8s native solution that satisfies category 3) can also take care of 1) and 2) while the native integration by itself can't achieve that. Existence of [1] might serve as

Re: Flink native k8s integration vs. operator

2022-01-04 Thread David Morávek
Hi Thomas, AFAIK there are no specific plans in this direction with the native integration, but I'd like to share some thoughts on the topic In my understanding there are three major groups of workloads in Flink: 1) Batch workloads 2) Interactive workloads (Both Batch and Streaming; eg. SQL

Flink native k8s integration vs. operator

2022-01-03 Thread Thomas Weise
Hi, I was recently looking at the Flink native Kubernetes integration [1] to get an idea how it relates to existing operator based solutions [2], [3]. Part of the native integration's motivations was simplicity (no extra component to install), but arguably that is also a shortcoming. The k8s