Re: (Possible) bug in flink-kafka-connector (metrics rewriting)

Hi, I suggest you creating a ticket for it on https://issues.apache.org/jira/projects/FLINK/summary. Flink - ASF JIRA Welcome to the Apache Flink project. Apache Flink is an open source platform for scalable batch and stream data processing.

Flink SQL and tumble window by size (number of rows)

Is it possible in Flink SQL to tumble a window by row size instead of time? Let's say that I want a window for every 1 rows for example using the Flink SQL API. is that possible? I can't find any documentation on how to do that, and I don't know if it is supported.

HybridSource permanently failed after restoring from checkpoint

Hi, folks I’m running Flink application that use HybridSource, patched with fixes FLINK-27479 and FLINK-27529 This application use HybridSource and presto plugin to read from a few thousands s3 directories, and then switch to reading from Kafka. Reading from s3 could cause intermittent

(Possible) bug in flink-kafka-connector (metrics rewriting)

Hello ! I would like to report a bug with metrics registration on KafkaProducer initialization. Firstly we found the problem with our Flink cluster: metric KafkaProducer.outgoing-byte-rate was periodically missing (was equals zero or near zero) on several subtasks, in the same time other

Re: WELCOME to user@flink.apache.org

Hey Karthikeyan, Welcome to Flink. Make sure that 1. IAM role has enough permissions to the required buckets. ( and /*) 2. 3. Once you define the IAM role, you need to annotate the SA with the role ARN and attach the SA to Flink pods. ("kubernetes.service-account" under

Failed to stop with savepoint

Env: Flink version:1.15.0 deploy mode :K8s applicaiton Mode. local mini cluster also have this problem. Kafka Connector : use Kafka SourceFunction . No new Api. action: I post a save with savepoint request to Flink Job throught rest api. A Error happened in Kafka connector close. The job will

Re: WELCOME to user@flink.apache.org

Hi Team, We have deployed our Flink Cluster on AWS EKS using Flink Operator. We have created required service accounts with IAM OIDC integration, however flink-main container seems to bypass this service account role and directly tries to create and delete objects in s3 using Node role. As the

Re: Flink Operator - delete flinkdeployments

Will do, thanks! On Tue, Aug 2, 2022 at 3:39 PM Gyula Fóra wrote: > Before trying to solve any already fixed problems please upgrade to 1.1.0 > :) > > > > On Tue, Aug 2, 2022 at 2:33 PM Sigalit Eliazov > wrote: > >> we are working with 1.0.0 >> >> On Tue, Aug 2, 2022 at 3:24 PM Gyula Fóra

Re: Flink Operator - delete flinkdeployments

Before trying to solve any already fixed problems please upgrade to 1.1.0 :) On Tue, Aug 2, 2022 at 2:33 PM Sigalit Eliazov wrote: > we are working with 1.0.0 > > On Tue, Aug 2, 2022 at 3:24 PM Gyula Fóra wrote: > >> Are you running the latest 1.1.0 version of the operator? >> >> Gyula >> >>

Re: Flink Operator - delete flinkdeployments

we are working with 1.0.0 On Tue, Aug 2, 2022 at 3:24 PM Gyula Fóra wrote: > Are you running the latest 1.1.0 version of the operator? > > Gyula > > On Tue, Aug 2, 2022 at 2:18 PM Sigalit Eliazov > wrote: > >> Hi, >> >> We are deploying a few flink clusters via the flink operator in our CI. >>

Re: Flink Operator - delete flinkdeployments

Are you running the latest 1.1.0 version of the operator? Gyula On Tue, Aug 2, 2022 at 2:18 PM Sigalit Eliazov wrote: > Hi, > > We are deploying a few flink clusters via the flink operator in our CI. > > In each run we first do a clean-up where one of the first steps is to > run 'kubectl

Flink Operator - delete flinkdeployments

Hi, We are deploying a few flink clusters via the flink operator in our CI. In each run we first do a clean-up where one of the first steps is to run 'kubectl delete flinkdeployments --all -n ' after that we also delete the flink operator pod and our all namespace. Lately we face issues where

Re: Migration to application mode

Thanks Lijie/Biao To put it simply, what we like to achieve is to replace env.runAsync() with some code that uses application mode. @Lijie: When I set deployment mode to application I got the following exception after submitting the job Caused by: java.lang.IllegalStateException: No

Re: Resources configuration on Kubernetes Session Cluster

Hi Vladislav! I am afraid there is no way to specify resources independently for jobs within a session cluster currently in Flink. For this I suggest using the Application Mode instead where each job can have its own resources. In any case you should check out the Flink Kubernetes Operator -

Resources configuration on Kubernetes Session Cluster

Hi, I'm trying to specify different TaskManager resources for different Flink jobs on Kubernetes Session Cluster. Can you help me to find a way to do that? I use this options, but Flink picks them up only when I start new Kubernetes Session deployment: Memory: jobmanager.memory.process.size

Re: Flink内部如何做到消息不丢失？

我最近也在对比storm和flink。有没有大佬介绍下，storm这种ack模式的是不是恢复会更快点，目前我感觉storm的架构下，各个节点的fail over更加独立感觉。 Flink 目前集群中任何一个机器失败都会导致整个任务重启，耗时会长点。 但是从全局资源来说，ckpt的资源占用貌似又比ack模式少。 不知道理解对不对。 tison 于2022年7月30日周六 14:28写道： > > 可以看下这两份材料 > > * >

Flink task lifecycle listener/hook/SPI

Hi all, We went to do some init env setup before the flink task run, And we have noticed the Task Lifecycle | Apache Flink doc described, but we can't find listener/hook/SPI interface do some custom init jobs