Re: Flink Kubernetes Operator podTemplate and 'app' pod label bug?

Hi! The app label itself is used by Flink internally for a different purpose so it’s overriden. This is completely expected. I think it would be better to use some other label :) Cheers, Gyula On Thu, 19 Jan 2023 at 19:02, Andrew Otto wrote: > Hello! > > I'm seeing an unexpected label value

Re: Regarding Changes in Flink Operator 1.3.1

Please see the release announcements: https://flink.apache.org/news/2022/10/07/release-kubernetes-operator-1.2.0.html https://flink.apache.org/news/2022/12/14/release-kubernetes-operator-1.3.0.html https://flink.apache.org/news/2023/01/10/release-kubernetes-operator-1.3.1.html

Re: Flink reactive mode for application clusters on AWS EMR - Auto Scaling

have any features or operators which we can use to run on Yarn > similar to Flink Kubernetes Operator which can perform auto-scaling. > > > Regards, > Madan > > On Friday, 13 January 2023 at 09:37:11 pm GMT-8, Gyula Fóra < > gyula.f...@gmail.com> wrote: > >

Re: Flink reactive mode for application clusters on AWS EMR - Auto Scaling

Hi Madan, With reactive mode you need to build a completely custom auto scaling logic, it can work but it takes considerable effort. Instead I recommend using the Flink Kubernetes Operator which now contains the first version of an actual autoscaler module that collects metrics and scales Flink

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

is > created while deploying the cluster at first place). > If yes, is this module can be used outside that operator? > > If not, would you please elaborate whether this scale(up/down) operation > leads to downtime? > > Best, > Tamir. > > > > > > > > >

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

nism ourselves on top of Flink clusters. > > Best, > Tamir. > -- > *From:* Gyula Fóra > *Sent:* Friday, January 13, 2023 8:39 AM > *To:* Swathi Chandrashekar > *Cc:* Chesnay Schepler ; Tamir Sagi < > tamir.s...@niceactimize.com>; u

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

ave standalone app mode cluster which is > managed by a flink Kubernetes operator and the operator would update the > replicas based on the metrics ( autoscale ) which in-tern changes the > parallelism as reactivemode is enabled. > > > > Regards, > > Swathi C > > > > *Fr

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

exactly the reactive mode provides vs what the operator does. Reactive mode alone doesn’t do too much. Gyula On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar wrote: > Hi @Gyula Fóra , > > > > Does this mean, with Kubernetes operator, we can have reactive mode in > native fli

Re: Flink reactive mode for application clusters on AWS EKS

Hey! I think the reactive scaling is a somewhat misunderstood feature. It only works in standalone deployments (not in Kubernetes native for instace) and it doesn't actually provide any autoscaling functionality on its own. You would have to implement your scaling logic yourself somehow

Re: Missing Flink Operator version

What you linked is not the list of stable releases but the list of all past releases. Only the last 2 minor versions are supported, which are currently 1.3.1 and 1.2.0. I highly recommend using the latest 1.3.1 release. Cheers, Gyula On Wed, Jan 11, 2023 at 7:31 AM Meghajit Mazumdar <

[ANNOUNCE] Apache Flink Kubernetes Operator 1.3.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.3.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

[ANNOUNCE] Apache Flink Kubernetes Operator 1.3.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.3.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

Re: What is the flink-kubernetes-operator webhook for?

To add to what Matyas said: Validation in itself is a mandatory step for every spec change that is submitted to guard against broken configs (things like negative parallelism etc). But validation can happen in 2 places. It can be done through the webhook, which would result in upfront rejection

Re: ArgoCD unable to process health from k8s FlinkDeployment - stuck in Processing

Hi! The resource lifecycle state is currently not shown explicitly in the status. You are confusing it with reconciliation status. At the moment you can only get this through the Java client:

Re: [Flink K8s Operator] flinkdep stays in DEPLOYED and never turns STABLE

Hi! The stable state is not marked in the reconciliation state field but instead using the last stable spec field. Deployed simply means that something is running :) The structure of the status is a bit complex to avoid too much redundancy and limit the size and is mostly considered to be

Re: Cleanup for high-availability.storageDir

Hi! There are some files that are not cleaned up over time in the HA dir that need to be cleaned up by the user: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/concepts/overview/#jobresultstore-resource-leak Hope this helps Gyula On Mon, 5 Dec 2022 at 11:56,

Re: flink-kubernetes-operator: image entrypoint misbehaves due to inability to write

here was a way to get > Secrets into the values used for rendering flink-conf.yaml too, so the > confs will be all in the same place. > > > > > > On Thu, Dec 1, 2022 at 9:30 AM Gyula Fóra wrote: > >> Andrew please see my previous response, that covers the secrets

Re: flink-kubernetes-operator: image entrypoint misbehaves due to inability to write

hem at WMF, but from a quick glance > I'm not sure how to combine them in flink-kubernetes-operator's ConfigMap > that renders flink-conf.yaml, but I feel like there should be a way. > > > > > On Wed, Nov 30, 2022 at 4:59 PM Gyula Fóra wrote: > >> Hi

Re: flink-kubernetes-operator: image entrypoint misbehaves due to inability to write

Hi Lucas! The Flink kubernetes integration itself is responsible for mounting the configmap and overwriting the entrypoint not the operator. Therefore this is not something we can easily change from the operator side. However I think we are looking at the problem from the wrong side and there may

Re: Savepoint restore mode for the Kubernetes operator

The operator might call dispose on an old savepoint that’s true, but I am not sure if the dispose api call would actually corrupt it. Gyula On Tue, 29 Nov 2022 at 09:28, Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Hangxiang, > > but, if I understand correctly, setting restore

Re: What is the recommended cpu.memory configuration of the Flink kubernetes operator in the production environment ?

We don't have a recommendation, this will depend on the number of jobs you manage. If you are managing a few jobs let's say < 50, the default resource configuration probably works well. The operator reports JVM metrics similar to Flink jobs so if you see longer GC pauses you could simply add more

Re: [Flink K8s operator] HA metadata not available to restore from last state

Hi Dongwon! This error mostly occurs when using Flink 1.14 and the Flink cluster goes into a terminal state. If a Flink job is FAILED/FINISHED (such as it exhausted the retry strategy), in Flink 1.14 the cluster shuts itself down and removes the HA metadata. In these cases the operator will only

Re: Deploy Flink Operator in an k8s enviroment without helm?

? > > > > *发件人:* user-return-51648-lifuqiong00=126@flink.apache.org > *代表 *Gyula Fóra > *发送时间:* 2022年11月18日 13:26 > *收件人:* Biao Geng > *抄送:* Mark Lee ; user@flink.apache.org > *主题:* Re: Deploy Flink Operator in an k8s enviroment without helm? > > > >

Re: Deploy Flink Operator in an k8s enviroment without helm?

Adding to what Biao Geng said, yes it is completely possible and other installation methods are used by many users already. You can check the Helm templates in the repo to get an idea what resources you need to create. Actually if you run `helm template flink-kubernetes-operator

Re: Owner reference with the Kubernetes operator

This has been changed in the current snapshot release: https://issues.apache.org/jira/browse/FLINK-28979 It will be part of the 1.3.0 version. On Wed, Nov 16, 2022 at 3:32 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hello, > > Is there a particular reason the operator doesn't

Re: support escaping `#` in flink job spec in Flink-operator

, Gyula Fóra wrote: > Hi! > > How do you submit your yaml? > > It’s possible that this is not operator problem. Did you try submitting > the deployment in json format instead? > > If it still doesn't work please open a JIRA ticket with the details to > reprod

Re: support escaping `#` in flink job spec in Flink-operator

Hi! How do you submit your yaml? It’s possible that this is not operator problem. Did you try submitting the deployment in json format instead? If it still doesn't work please open a JIRA ticket with the details to reproduce and what you have tried :) Cheers Gyula On Tue, 8 Nov 2022 at 04:56,

Re: configMap value error when using flink-operator?

flink-conf.yaml of Kubernetes operator to 1, which are aligned with >> the default values in Flink codebase. >> >> >> Best, >> Yang >> >> Gyula Fóra 于2022年10月26日周三 15:17写道： >> >>> Hi! >>> >>> I agree that this might be confu

Re: configMap value error when using flink-operator?

Hi! I agree that this might be confusing but let me explain what happened. In the operator you can define default flink configuration. Currently it is https://github.com/apache/flink-kubernetes-operator/blob/main/helm/flink-kubernetes-operator/conf/flink-conf.yaml It contains

Re: status no clear when deploying batch job with flink-k8s-operator

When a batch job finishes and the cluster is shut down the operator cannot observe the status. It is impossible to tell whether it finished or not. Try upgrading to Flink 1.15, there this is solved. Cheers, Gyula On Tue, Oct 25, 2022 at 9:23 AM Liting Liu (litiliu) wrote: > Hi, I'm deploying

Re: Flink Native K8S RBAC

Hi! As a reference you can look at how the Flink Kubernetes Operator manages RBAC settings: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/

Re: Does kubernetes operator support manually triggering savepoint with canceling the job?

I think you are confusing manual savepoints with savepoint upgrades. Manual savepoints will trigger a savepoint but not shut down the job. If you want to stop the job with savepoint you set the upgradeMode to savepoint and set the state to SUSPENDED.

Re: Activate Flink HA without checkpoints on k8S

Without HA, if the jobmanager goes down, job information is lost so the job won’t be restarted after the JM comes back up. Gyula On Thu, 13 Oct 2022 at 19:07, marco andreas wrote: > > > Hello, > > Can someone explain to me what is the point of using HA when deploying an > application cluster

Re: HA not working in standalone mode for operator 1.2

Before we dive further into this can you please explain the jarURI problem your are trying to solve by switching to standalone? The native mode should work well in almost any setup. Gyula On Thu, 13 Oct 2022 at 21:41, Javier Vegas wrote: > Hi, I have a S3 HA Flink app that works as expected

Re: Kubernetes operator assign Job ID

Hi! This change aligns with how newer Flink 1.16+ versions handle application job ids. There are some good reasons for doing this please see: https://issues.apache.org/jira/browse/FLINK-19358 https://issues.apache.org/jira/browse/FLINK-29109 If you want to go back to the old behaviour you need

Re: Job uptime metric in Flink Operator managed cluster

er-v7] Resource fully reconciled, > nothing to do... > > Also, my original doubt was around the uptime metric itself. What is the > correct metric to use for monitoring the status ( running or suspended) of > a job which is being managed by the Flink Operator ? > The *jobmana

Re: allowNonRestoredState doesn't seem to be working

Hi! If you have last-state upgrade mode configured it may happen that the allowNonRestoredState config is ignored by Flink (as the last-state upgrade mechanism somewhat bypasses the regular submission). Worst case scenario, you can suspend the deployment, manually record the last

Re: Job uptime metric in Flink Operator managed cluster

Hello! The Flink operator currently does not delete the jobmanager pod when a deployment is suspended. This way the rest api stay available but no other resources are consumed (taskmanagers are deleted) When you delete the FlinkDeployment resource completely, then the jobmanager deployment is

Re: videos Flink Forward San Francisco 2022

I think everyone would be happier with the videos published on Youtube but it's unfortunately at the discretion of the organizer. At this time they decided against it for some reason. Gyula On Mon, Oct 10, 2022 at 11:57 AM Martin wrote: > Hey, > > that's sad. Is it possible for future Flink

Re: Validation error trying to use standalone mode with operator 1.2.0

Hi! Seems like you still have an older version CRD installed for the FlinkDeployment which doesn’t contain the newly introduced mode setting. You can check https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.2/docs/operations/upgrade/ for the upgrade process. Cheers

[ANNOUNCE] Apache Flink Kubernetes Operator 1.2.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.2.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

[ANNOUNCE] Apache Flink Kubernetes Operator 1.2.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.2.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

Re: JobManager restarts on job failure

ispatcher.java#L663 > [2] https://issues.apache.org/jira/browse/FLINK-29415 > [3] > https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-core/src/main/java/org/apache/flink/configuration/PipelineOptionsInternal.java#L29 > > On Tue, Sep 20, 2022 at 3:45 PM

Re: JobManager restarts on job failure

lication mode cluster). > > > > ------ > *От:* Gyula Fóra > *Отправлено:* 20 сентября 2022 г. 19:49:37 > *Кому:* Evgeniy Lyutikov > *Копия:* user@flink.apache.org > *Тема:* Re: JobManager restarts on job failure > > The best thing for you to do would

Re: JobManager restarts on job failure

The best thing for you to do would be to upgrade to Flink 1.15 and the latest operator version. In Flink 1.15 we have the option to interact with the Flink jobmanager even after the job FAILED and the operator leverages this for a much more robust behaviour. In any case the operator should not

Re: Deploying Jobmanager on k8s as a Deployment

la, how are JMs handled in the operator? Job, Deployment, or something > custom? > > > Best, > Austin > > > > On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: > >> You can use deployments of course , the operator and native k8s >> integration does exactly tha

Re: Deploying Jobmanager on k8s as a Deployment

will take a look at it. > > But if we aren't able to use the flink operator due to technical > constraints is it possible to deploy the JM as deployment without any > consequences that I am not aware of? > > Sincerely, > > Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a écrit :

Re: Deploying Jobmanager on k8s as a Deployment

Hi! You should check out the Flink Kubernetes Operator. I think that covers all your needs . https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ Cheers, Gyula On Sat, 3 Sep 2022 at 13:45, marco andreas wrote: > > We are deploying a flink application cluster on k8S.

Re: start flink-operator using "watchNamespaces"

Hi! >From your logs I see: Configuring operator to watch the following namespaces: [JOSDK_ALL_NAMESPACES] This means that the operator is not configured correctly for the watched namespaces. After you change the helm values you might have to reinstall the operator. Cheers, Gyula On Mon, Aug

Re: Savepoint problen on KubernetesOperator HA cluster

In general the Flink JobManager HA /client mechanism ensures that the rest requests end up at the current leader. In your case it's not clear what the actual cause of the issue was. What I would do is to upgrade to the latest operator version (1.1.0) where the savepoint upgrade mechanism has

Re: Flink Operator - delete flinkdeployments

, 2022 at 3:39 PM Sigalit Eliazov > wrote: > >> Will do, thanks! >> >> On Tue, Aug 2, 2022 at 3:39 PM Gyula Fóra wrote: >> >>> Before trying to solve any already fixed problems please upgrade to >>> 1.1.0 :) >>> >>> >>> >&g

Re: Flink Operator - delete flinkdeployments

Before trying to solve any already fixed problems please upgrade to 1.1.0 :) On Tue, Aug 2, 2022 at 2:33 PM Sigalit Eliazov wrote: > we are working with 1.0.0 > > On Tue, Aug 2, 2022 at 3:24 PM Gyula Fóra wrote: > >> Are you running the latest 1.1.0 version of the ope

Re: Flink Operator - delete flinkdeployments

Are you running the latest 1.1.0 version of the operator? Gyula On Tue, Aug 2, 2022 at 2:18 PM Sigalit Eliazov wrote: > Hi, > > We are deploying a few flink clusters via the flink operator in our CI. > > In each run we first do a clean-up where one of the first steps is to > run 'kubectl

Re: Resources configuration on Kubernetes Session Cluster

Hi Vladislav! I am afraid there is no way to specify resources independently for jobs within a session cluster currently in Flink. For this I suggest using the Application Mode instead where each job can have its own resources. In any case you should check out the Flink Kubernetes Operator -

[ANNOUNCE] Apache Flink Kubernetes Operator 1.1.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.1.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

[ANNOUNCE] Apache Flink Kubernetes Operator 1.1.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.1.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for

Re: Guide for building Flink image with Python doesn't work

[1] https://pypi.org/project/pemja/0.1.4/ > [2] https://issues.apache.org/jira/browse/FLINK-25188 > > Gyula Fóra 于2022年7月6日周三 13:56写道： > >> Here it is, copied from the docs essentially: >> >> FROM flink:1.15.0 >> >> >> # install python3: it has updated Python t

Re: Difference between Session Mode and Session Job(Flink Opearator)

Hi! The Flink Kubernetes Operator on a high level supports 3 types of resources: 1. Session Deployment : Empty Flink Session cluster 2. Application Deployment: Flink Application cluster (single job / cluster) 3. Session Job: Flink Job deployed to an existing Session Deployment. So

Re: Guide for building Flink image with Python doesn't work

g/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker > > Best, > Weihua > > > On Tue, Jul 5, 2022 at 11:40 PM Gyula Fóra wrote: > >> Well in any case either the official image is incorrect (maybe we should &g

Re: Best practice for creating/restoring savepoint in standalone k8 setup

look but it would also be > interesting to see how this have been solved before the introduction of the > Flink operator > > Den tis 5 juli 2022 kl 16:37 skrev Gyula Fóra : > >> Hi! >> >> Did you check the https://github.com/apache/flink-kubernetes-operator &g

Re: Guide for building Flink image with Python doesn't work

cial image: >> >> >> https://hub.docker.com/layers/openjdk/library/openjdk/11-jdk/images/sha256-bc0af19c7c4f492fe6ff0c1d1c8c0e5dd90ab801385b220347bb91dbe2b4094f?context=explore >> >> >> Best, >> Weihua >> >> >> On Tue, Jul 5, 2022 at 3:50 PM Gy

Re: Best practice for creating/restoring savepoint in standalone k8 setup

Hi! Did you check the https://github.com/apache/flink-kubernetes-operator by any chance? It provides many of the application lifecycle features that you are probably after straight out-of-the-box. It has both manual and periodic savepoint

Guide for building Flink image with Python doesn't work

Hi All! I have been trying to experiment with the Flink python support on Kubernetes but I got stuck creating a custom image with all the necessary python libraries. I found this guide in the docs:

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.1 released

Sorry for the incorrect blogpost link, please ignore that. There is no blogpbost for 1.0.1 :) Gyula On Tue, Jun 28, 2022 at 9:43 AM Yang Wang wrote: > Thanks Gyula for working on the first patch release for the Flink > Kubernetes Operator project. > > > Best, > Yang > >

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.1 released

Sorry for the incorrect blogpost link, please ignore that. There is no blogpbost for 1.0.1 :) Gyula On Tue, Jun 28, 2022 at 9:43 AM Yang Wang wrote: > Thanks Gyula for working on the first patch release for the Flink > Kubernetes Operator project. > > > Best, > Yang > >

[ANNOUNCE] Apache Flink Kubernetes Operator 1.0.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.0.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl.

[ANNOUNCE] Apache Flink Kubernetes Operator 1.0.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.0.1. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl.

Re: Flink k8s Operator on AWS?

A small addition to what Matyas has said: The limitation of only supporting local scheme is coming from the Flink Kubernetes Application mode directly and is not related to the operator itself. Once this feature is added to Flink itself the operator can also support it for newer Flink versions.

Re: Flink operator deletes the FlinkDeplyoment after a while

Hi Sigalit, This could be related to https://issues.apache.org/jira/browse/FLINK-27889 We have fixed this issue already (after the release), you could simply use the latest operator image from of `release-1.0: *ghcr.io/apache/flink-kubernetes-operator:cc8207c

Re: Flink Operator 1.0.0 not working

Seems like something is off with your CRD. You could try replacing it using: kubectl replace -f https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.0/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml After that you should be able to deploy the

Re: Flink Kubernetes Operator v1.0 ETA

Hi Jeesmon! We are currently working through the release process. We are now in the middle of voting for RC4 (we have identified and fixed a number of blocker issues in earlier RCs). We are hopeful that this RC will be successful in which case you will have a release by the end of the week. If

Re: Python Job Type Support in Flink Kubernetes Operator

Hi Jeesmon! Sorry I completely missed this question earlier :) There is no support currently for Python jobs and I don't really have any experience with Python jobs so cannot really comment on how easy it would be to integrate it. We and most of the companies currently involved with developing

Re: Flink Kubernetes Operator: Specifying env variables from ConfigMaps

Hi Mads! I think you need to use the podTemplate for this. You can either do it in the top level spec or customize it for tm/jm respectively. Keep in mind that pod templates are merged with the base flink template so it's enough to specify the fields relevant for you (in these case the env

Re: Example of Beam Go job that uses Flink Kubernetes operator

Hi Red! Thank you for your interest and for checking out the official Flink Kubernetes Operator Project. This project is still relatively new and so far our focus has been on running native Flink workloads (Flink Application/Session clusters, and session jobs) and we haven't tried running a

Re: Flink Kubernetes operator not having a scale subresource

On Sat, May 7, 2022 at 11:25 AM Gyula Fóra wrote: > Hi Jay! > > I will take a closer look into this and see if we can use the parallelism > in the scale subresource. > > If you could experiment with this and see if it works with the current CRD > that would be helpful

Re: Flink Kubernetes operator not having a scale subresource

.apache.org/confluence/display/FLINK/FLIP-225%3A+Implement+standalone+mode+support+in+the+kubernetes+operator > > Best, > Yang > > Gyula Fóra 于2022年5月7日周六 04:26写道： > >> Hi Jay! >> >> Interesting question/proposal to add the scale-subresource. >> >> I

Re: Flink Kubernetes operator not having a scale subresource

Hi Jay! Interesting question/proposal to add the scale-subresource. I am not an expert on this area but we will look into this a little and give you some feedback and see if we can incorporate something into the upcoming release if it makes sense. On a high level there is not a single replicas

Re: Kubernetes Operator --allowNonRestoredState config

Hi! We did not expose this as a top level flag in the spec, but you can enable this through the flink configuration using: execution.savepoint.ignore-unclaimed-state: "true" Cheers, Gyula On Fri, Apr 29, 2022 at 5:26 PM Shqiprim Bunjaku wrote: > Hi all, > > I am using Flink Kubernetes

Re: Enabling savepoints when deploying in Application Mode

Hi Lilli! I am not aware of any problems with savepoint restore in application mode. What you can try is to use the *execution.savepoint.path *configuration setting to control it instead of the CLI and see if it makes a difference for you. Otherwise, you could also check out the

[ANNOUNCE] Apache Flink Kubernetes Operator 0.1.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 0.1.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. This is the first release of this new

[ANNOUNCE] Apache Flink Kubernetes Operator 0.1.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 0.1.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. This is the first release of this new

Re: The flink-kubernetes-operator vs third party flink operators

Hi! The main difference at the moment is the programming language and the APIs used to interact with Flink. The flink-kubernetes-operator, uses Java and interacts with Flink using the built in (native) clients. The other operators have been around since earlier Flink versions. They all use

Re: Helm install flink-kubernetes-operator failed

As Biao Geng said this will be much easier after the first preview release. Which should become available on monday if all works out :) Until then you can also test our last release candidate which will hopefully become the release: helm repo add operator-rc3

Re: flink cluster startup time

you will have some time to test our preview release and share some feedback! Cheers, Gyula On Thu, Mar 31, 2022 at 4:51 AM Yang Wang wrote: > @Gyula Fóra is trying to prepare the preview > release(0.1) for flink-kubernetes-operator. It now is fully functional for > application mo

Re: Deploy a Flink session cluster natively on K8s with multi AZ

Hi! I think the Flink Kubernetes Operator ( https://github.com/apache/flink-kubernetes-operator) project is exactly what you are looking for. This is a relatively new addition to Flink that supports k8s application and session deployments with lifecycle management through kubernetes native

Re: Where can I find flink-kubernetes-operator project?

Hi Makas! The repo is here: https://github.com/apache/flink-kubernetes-operator We should add a link somewhere in the docs :) Cheers, Gyula On Fri, Mar 25, 2022 at 9:45 AM Makas Tzavellas wrote: > Hello, > > I came across this document >

Re: Flink native k8s integration vs. operator

;>> late. > >>> >> > > >>> >> > I agree with Thomas' and David's assessment of Flink's "Native > >>> >> Kubernetes > >>> >> > Integration", in particular, it does actually not integrate well

Re: Flink native k8s integration vs. operator

Hi All! This is a very interesting discussion. I think many users find it confusing what deployment mode to choose when considering a new production application on Kubernetes. With all the options of native, standalone and different operators this can get tricky :) I really like the idea that

Re: [ANNOUNCE] Dropping "CheckpointConfig.setPreferCheckpointForRecovery()"

Hi Stephan, I do not know if anyone is still relying on this but I think it makes sense to drop this feature. So +1 from me. I think it served a valid purpose originally but if we have a good improvement in the pipeline using the savepoints directly that will solve the problem properly. I would

Re: Using double quotes for SQL identifiers

t; > > > AFAIK, except the sql-dialect, table API does not expose any parser > > related configuration to the user. > > But we still can change the config of quoting identifiers in parser with > > some code changing. > > You can reference this test class: &

Using double quotes for SQL identifiers

Hi All! Is it possible in any way to configure the Table environments to allow double quotes (") to be used for identifiers instead of backticks (`). Thank you! Gyula

Re: [DISCUSS] Upgrade HBase connector to 2.2.x

tuff into Flink, and legacy, >> experimental or unstable connectors into Bahir. >> >> >> Who can take care of this effort? (Decide which Hbase 2 PR to take, >> review and contribution to Bahir) >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-18795 &

Re: [DISCUSS] Upgrade HBase connector to 2.2.x

If we were to go the bahir route, I don't see the point in migrating the 1.4.x version there since that's already available in Flink. To me that is almost the same as dropping explicit support for 1.4 and telling users to use older connector versions if they wish to keep using it. If we want to

Re: Re: [DISCUSS] Upgrade HBase connector to 2.2.x

d user-zh@ to hear more feedback from users. > > Best, > Jark > > On Thu, 18 Jun 2020 at 21:25, Gyula Fóra wrote: > >> Hi All! >> >> I would like to revive an old ticket >> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around >&g

Re: Using logicalType in the Avro table format

ma being correctly > translated to a specific record that uses the new TimeConversions [1]. > > [1] > https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/data/TimeConversions.java > > On Thu, Apr 30, 2020 at 10:41 AM Gyula Fóra wrote: > >&

Re: Using logicalType in the Avro table format

ost/not respected the planner produces LocalDateTime > instead of a proper java.sql.Timestamp time. The AvroRowSerializationSchema > expects java.sql.Timestamp for a column of TIMESTAMP type and thus it fails > for LocalDateTime. I really hope the effort of FLIP-95 will significantly > reduce t

Using logicalType in the Avro table format

Hi All! We are trying to work with avro serialized data from Kafka using the Table API and use TIMESTAMP column type. According to the docs , we can use long type with logicalType:

Re: Converting String/boxed-primitive Array columns back to DataStream

k for you? The other methods take > TypeInformation and might cause this problem. It is definitely a bug. > > Feel free to open an issue under: > https://issues.apache.org/jira/browse/FLINK-12251 > > Regards, > Timo > > On 28.04.20 18:44, Gyula Fóra wrote: > > Hi Timo,

Re: Converting String/boxed-primitive Array columns back to DataStream

ng between old TypeInformation and new DataType > system. A back and forth conversion should work between those types. > > Regards, > Timo > > On 28.04.20 15:36, Gyula Fóra wrote: > > Hi All! > > > > I have a Table with columns of ARRAY and ARRAY, is there &g

Converting String/boxed-primitive Array columns back to DataStream

Hi All! I have a Table with columns of ARRAY and ARRAY, is there any way to convert it back to the respective java arrays? String[] and Integer[] It only seems to work for primitive types (non null), date, time and decimal. For String for instance I get the following error: Query schema: [f0:

<    1   2   3   4   >