Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

Thank you, Gyula! 🥳 Cheers On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.9.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink > applications and their lifecycl

Re: Checkpoint jitter?

e with the > running updates. > > What you get is a little more overall I/O but a quite flat I/O rate. > > > > > > Hope this helps > > > > Thias > > > > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/s

Re: Checkpoint jitter?

same time. > > Gyula > > On Wed, Sep 13, 2023 at 10:32 AM Hangxiang Yu wrote: > >> Hi, Matyas. >> Do you mean something like adjusting checkpoint intervals dynamically >> or frequency of uploading files according to the pressure of the durable >> storage ? >> >

Checkpoint jitter?

Hey folks, Is it possible to add some sort of jitter to the checkpointing logic for massively parallel jobs to mitigate the burst impact on the durable storage when a checkpoint is triggered? Thanks, Matyas

Custom source function in PyFlink

Hey folks, I'm looking for an example for creating a custom source in PyFlink. The one that I found in the tests is a wrapper around a java class: def test_add_custom_source(self): custom_source = SourceFunction( "org.apache.flink.python.util.MyCustomSourceFunction") ds = self.env.add_source(cust

Re: Flink K8s operator pod section of CRD

Yep! Simple oversight, it was :/ Cheers, Matyas On Thu, Feb 23, 2023 at 10:54 PM Gyula Fóra wrote: > Hey! > You are right, these fields could have been of the PodTemplate / > PodTemplateSpec type (probably PodTemplateSpec is actually better). > > I think the reason why we used it is two fold:

Re: Flink Kubernetes Operator podTemplate and 'app' pod label bug?

On a side note, we should probably use a qualified label name instead of the pretty common app here. WDYT Gyula? On Thu, Jan 19, 2023 at 1:48 PM Gyula Fóra wrote: > Hi! > > The app label itself is used by Flink internally for a different purpose > so it’s overriden. This is completely expected.

Re: Certificate rotation when using SSL

We've added something similar to the FKO: https://github.com/apache/flink-kubernetes-operator/pull/364 Best, Matyas On Fri, Dec 16, 2022 at 5:11 AM Martijn Visser wrote: > Hi Steve, > > I don't think that Flink has added support for certificate rotation. It > would be quite a nice feature if so

[ANNOUNCE] Apache Flink Kubernetes Operator 1.3.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.3.0. Release highlights: - Upgrade to Fabric8 6.x.x and JOSDK 4.x.x - Restart unhealthy Flink clusters - Contribute the Flink Kubernetes Operator to OperatorHub - Publish flink-kuber

Re: What is the flink-kubernetes-operator webhook for?

Hi Otto, webhooks in general are optional components of the k8s operator pattern. Mostly used for validation, sometimes for changing custom resources and handling multiple versions, etc. It's an optional component in the Flink Kubernetes Operator too. Regards, Matyas On Fri, Dec 9, 2022 at 5:59

Re: Apache vs Spotify Flink Operator?

Hi Clayton, We recently discussed these topics with Robert and Gyula in this podcast. Cheers, Matyas On Wed, Nov 23, 2022 at 2:45 PM Clayton Wohl wrote: > At my job, we are using the Spotify Flink Operator in production. Are > there any pros/cons o

Re: Does kubernetes operator support manually triggering savepoint with canceling the job?

This should do the trick: job: upgradeMode: savepoint state: suspended In the CR you should see something similar, after applying the above change. lastSavepoint: formatType: CANONICAL location: file:/flink-data/savepoints/savepoint-fc61e1-b9cf089c260e timeStamp

Re: Understanding webhook checks in flink operator

Hi Prakhar, the webhook is an optional component in the Flink Operator. It's responsible mainly for validation through: https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/validation/DefaultValidator.java This

Re: start flink-operator using "watchNamespaces"

Also you can check if the configs are updated properly in the ConfigMap. It should be auto-populated when you reinstall it from Helm as Gyula suggested. *kubernetes.operator.watched.namespaces: ns1, ns2, ...* On Tue, Aug 23, 2022 at 5:37 PM Gyula Fóra wrote: > Hi! > > From your logs I see: > > C

Re: Flink Operator Resources Requests and Limits

gt; >> cpu: "2" >> >> memory: "2048m" >> >> >> >> results in a pod like this: >> >> Limits: >> >> cpu: 1 >> >> memory: 1600Mi >> >> Reque

Re: Flink Operator Resources Requests and Limits

Hi James, Have you considered using pod templates already? https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/ Regards, Matyas On Wed, Jul 27, 2022 at 3:21 PM PACE, JAMES wrote: > We are currently evaluating the apache flink operator (versi

Re: Kubernetes Operator - Logging

Hi Daniel, Some additional info on the current approach we have in the operator: Manipulating plain property files was relatively easy from the Helm chart, hence we went down this route first. We could try to document at least an example for logback then see if we can improve the setup further.

Re: Flink k8s Operator on AWS?

ava 8 > being used, indicated in the log. > > org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - > JAVA_HOME: /usr/local/openjdk-8 > > Is there a way to use Java 11 to start Flink with? > > Kind regards, > > Matt > > On Tue, Jun 21, 2022 at 4:53

Re: Advice needed: Flink Kubernetes Operator with Prometheus Configuration

Hi Morgan, There is a placeholder in the values.yaml: [image: image.png] This should create an entry on the operator container: [image: image.png] Can you share the output of this command pls? k get deploy flink-kubernetes-operator -o yaml Thanks, Matyas On Thu, Jun 23, 2022 at 12:12 PM Ge

Re: Flink operator - ignore ssl cert validation

I guess the problem is caused by the self-signed certificates, and Calvin is looking for a way to disable the SSL certificate validation: something like this http://www.java2s.com/Code/Java/Network-Protocol/DisablingCertificateValidationinanHTTPSConnection.htm On Thu, Jun 23, 2022 at 11:40 AM Yang

Re: Flink Operator - Support for k8s HA jobmanager

I guess the problem here is that your JM pods do not have access to a common upload folder. You need to mount a shared volume for all the JobManagers and point to it using the 'web.upload.dir: property. On Wed, Jun 22, 2022 at 8:45 PM calvin beloy wrote: > Hi, > > Trying to deploy FlinkSessionJ

Re: Flink k8s Operator on AWS?

ient (running kubectl) to the master? If so, > could we add an option to the job specification to that effect? Just > brainstorming ;-) (and forking apache/flink-kubernetes-operator) > > All the best, > Matt > > On Tue, Jun 21, 2022 at 2:52 PM Őrhidi Mátyás > wrote: > >>

Re: Flink k8s Operator on AWS?

Hi Matt, - In FlinkDeployments you can utilize an init container to download your artifact onto a shared volume, then you can refer to it as local:/.. from the main container. FlinkDeployments comes with pod template support https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/do

Re: Flink Kubernetes Operator with K8S + Istio + mTLS - port definitions

It seems Istio must be configured to allow Akka cluster communication to bypass the Istio sidecar proxy: https://doc.akka.io/docs/akka-management/current/bootstrap/istio.html On Mon, Jun 20, 2022 at 11:30 AM Sigalit Eliazov wrote: > Hi, > we have enabled HA as suggested, the task manager tries t

Re: Question of Flink Operator Application Cluster Deployment

You don't have to mount the service account explicitly, this should be auto-mounted for you. Please share your (redacted) yamls for the RBAC configs ( https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-othe

Re: flink operator sometimes cannot start jobmanager after upgrading

I filed a Jira for tracking this issue: https://issues.apache.org/jira/browse/FLINK-27468 On Mon, May 2, 2022 at 10:31 AM Őrhidi Mátyás wrote: > This can be reproduced simply by deleting the kubernetes deployment. The > operator cannot recover from this state automatically, by defi

Re: flink operator sometimes cannot start jobmanager after upgrading

This can be reproduced simply by deleting the kubernetes deployment. The operator cannot recover from this state automatically, by defining a restartNonce on the deployment should recover the state. Regards, Matyas On Mon, May 2, 2022 at 10:00 AM Márton Balassi wrote: > Hi ChangZhuo, > > Thanks

Re: Using the official flink operator and kubernetes secrets

Also, just declaring it in the flink configs should be sufficient, no need to define it in the pod templates: flinkConfiguration: kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" Best, Matyas On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mát

Re: Using the official flink operator and kubernetes secrets

Hi Francis, I suggest accessing the environment variables directly, no need to pass them as command arguments I guess. Best, Matyas On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy < francis.con...@switchdin.com> wrote: > Hi all, > > I'm trying to use a kubernetes secret as a command line argume

Re: flink-kubernetes-operator: Flink deployment stuck in scheduled state when increasing resource CPU above 1

It's worth checking the deployment->replica set->pod chain for error message. On Fri, Mar 25, 2022, 19:49 Makas Tzavellas wrote: > Hi, > > I have been experimenting with flink-kubernetes-operator today and it's > very cool. However, the Flink application will be stuck in scheduled state > if I i

Testing jobs locally agains secure Hadoop cluster

Dear Community, I'm having troubles testing jobs against a secure Hadoop cluster. Is that possible? The mini cluster seems to not load any security modules. Thanks, Matyas

Re: table.show() in Flink

gt;> cc @Godfrey, please correct if I misunderstand the above API. >> >> Best, >> Jark >> >> [1]: >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878 >> >> On Tue, 5 May 2020 at 20:19, Őrhidi Mátyás >> wrote: >>

table.show() in Flink

Dear Flink Community, I'm missing Spark's table.show() method in Flink. I'm using the following alternative at the moment: Table results = tableEnv.sqlQuery("SELECT * FROM my_table"); tableEnv.toAppendStream(results, Row.class).print(); Is it the recommended way to print the content of a table?

define WATERMARKS in queries/views?

Dear Community, is it possible to define WATERMARKS in SQL queries/views? We have a read only catalog implementation and we would like to assign WMs to the tables somehow. Thanks, Matyas