Re: Problems sinking a Flink table to Kafka

Hi, You are using a special Kafka connector. From the definition in website: "as a sink, the upsert-kafka connector can consume a changelog stream. It will write INSERT/UPDATE_AFTER data as normal Kafka messages value, and write DELETE data as Kafka messages with null values (indicate tombstone

Re: Job Manager getting restarted while restarting task manager

> class used by jobmanager only. > > Best, > Xintong > > > > On Thu, Oct 13, 2022 at 9:06 AM yu'an huang wrote: > >> Hi, >> >> Which deployment mode do you use? What is the Flink version? >> I think killing TaskManagers won't make

Re: Job Manager getting restarted while restarting task manager

Hi, Which deployment mode do you use? What is the Flink version? I think killing TaskManagers won't make the JobMananger restart. You can provide the whole log as an attachment to investigate. On Wed, 12 Oct 2022 at 6:01 PM, Puneet Duggal wrote: > Hi Xintong Song, > > Thanks for your immediate

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

I guess there may be a class conflict between your user jar and Flink lib. For the error: java.lang.NoClassDefFoundError, it may caused Exception happening when Initializing a class. I suggest you set the log level to DEBUG and send the client log here. Let’s look whether there are any new findi

The configuration might be lost in application mode when JobManager restarts

Hi team, We found a case that the job configuration would be lost in application mode if the job manager restarted. When developing a job, users might want to set their configuration in the main method of their user program. This is fine for YARN per job mode. The client will run the user pro

Re: Running Flink Spring boot application on Kubernetes

Hi, Which class it can not be found? How do you write your spring boot application? Can you also share more logs about that? Best, Yuan On Wed, 24 Aug 2022 at 3:38 PM, Yogesh Jadhav wrote: > Hi, > If anyone has any thoughts on this. I have created Java Spring boot > application which is workin

Re: unsubscribe

Hi, To remove your address from the list, send a message to: On Sun, 21 Aug 2022 at 9:55 PM, Venkatesh kalyan wrote: > want to unsubscribe . >

Re: unsubscribe

Hi，To remove your address from the list, send a message to: On Mon, 22 Aug 2022 at 4:08 PM, Sijun Yang wrote: >

Re: unsubscribe

Hi， To remove your address from the list, send a message to: On Mon, 22 Aug 2022 at 1:42 PM, Alex Drobinsky wrote: > Please unsubscribe me >

Re: Pyflink :: Conversion from DataStream to TableAPI

xxx().force_non_parallel() > ``` > > Regards, > Dian > > On Wed, Aug 17, 2022 at 11:13 AM Ramana wrote: > >> Hi Yuan - Thanks for your response. Wondering if the window api supports >> non-keyed streams? >> >> On Wed, Aug 17, 2022, 06:43 yu'

Re: Pyflink :: Conversion from DataStream to TableAPI

Hi, Pyflink should support window api. You can read this document. https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/datastream/operators/windows/ Hope this helps. Best, Yuan On Tue, 16 Aug 2022 at 3:11 PM, Ramana wrote: > Hi All - > > Trying to achieve the following

Re: Metrics OOM java heap space

s is that the problem is not in metrics content, but in their > number, which increases with the number of operators. > So the next question is if there is a way to exclude metric generation on > operator level. > Found same question without correct answer on SOF: > https://stackoverfl

Re: Metrics OOM java heap space

Hi Yuriy, How do you set your TaskMananger Memory? I think 40MB is not significant high for Flink. And It’s normal to see memory increase if you have more parallelism or set another metrics on. You can try setting larger moratory for Flink as explained by following documents. https://nightlie

Re: Flink TaskManager memory configuration failed

Hi John, May I know what is your Flink version you are trying? On Thu, 23 Jun 2022 at 3:43 AM, John Tipper wrote: > Hi all, > > I'm wanting to run quite a number of PyFlink jobs on Kubernetes, where the > amount of state and number of events being processed is small and therefore > I'd like t

Re: HTTP 404 while creating resource with flink kubernetes operator and frabric8 client

Hi， It seems that you can't find the FlinkDeployment. I saw the api server address is maskedip. Can you confirm whether it is the correct api server address? Best, Yuan On Thu, 23 Jun 2022 at 11:52 AM, Kishore Pola wrote: > Hello flink user group, > > When I am trying to create a flink deploym

Re: Savepoint (with job cancel) while checkpoint in progress

innumerable 'job cancels' during deployments and we have never seen anything > like the sequence above. So, it's very odd. > > Thanks > Sudharsan > > > On Sun, Jun 19, 2022 at 2:22 AM yu'an huang <mailto:h.yuan...@gmail.com>> wrote: > Hi S

Re: Savepoint (with job cancel) while checkpoint in progress

Hi Sudharsan, How did you cancel thus single job. According to the High Availability Document: “In order to recover submitted jobs, Flink persists metadata and the job artifacts. The HA data will be kept until the respective job either succeeds, is cancelled or fails terminally. Once this ha

Re: Could not copy native libraries - Permission denied

What is your deployment mode, on yarn, Kubernetes or standalone? Can you provide more logs about this error? > On 18 May 2022, at 4:07 PM, Zain Haider Nemati wrote: > > Hi, > We are using flink version 1.13 with a kafka source and a kinesis sink with a > parallelism of 3. > On submitting the

Re: Applying backpressure to limit state memory consumption

H Robini, In my experience, the state size of memory state backend is limit by the heap memory. See this link for details: https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/ “When deciding between HashMapStateBackend and RocksDB, it is a choice between performan

Re: Incompatible data types while using firehose sink

Hi, Your code is working fine in my computer. What is the Flink version you are using. > On 12 May 2022, at 3:39 AM, Zain Haider Nemati wrote: > > Hi Folks, > Getting this error when sinking data to a firehosesink, would really > appreciate some help ! > > DataStream inputStream = env.ad

Re: Flink EOF Error on Running Code

Hi, It throws this exception because the memory segment provided by Flink is depleted. How do you implement your writing logic? It seems that you are using apache beam. What is the version of Flink engine? Can you provide more logs so we can know why the memory segment depletion happened. I

Re: NO jobmanager.log for yarn-per-job submitted with sql-client.sh

Hi ruiyun, Is there any log4j configuration file or logback.xml in your Flink configuration directory? Yarn should check whether the file exists first and set relative log options. > On 26 Apr 2022, at 5:21 PM, ruiyun wan wrote: > > Flink Version (1.13) > There is not jobmanager.log file

Re: How to dynamically modify the schema information of a table

Thank you for your reporting. I created a ticket in JIRA. You can track this problem in this link: https://issues.apache.org/jira/browse/FLINK-27449 . > On 22 Apr 2022, at 10:44 AM, 草莓 wrote: > > The following is the Java code > > @Test >

Re: Flink state migration from 1.9 to 1.13

Hi Qinghui, Did you used a difference keyby() for your KeyedCoProcesserOperator? For example, did you use a fied name (keyBy(“id”)) in 1.9 and while use a lambda (keyBy(e->e.getId()) in 1.13. This will make the key serializer incompatible. You may reference this link for how to use Apache Flink

Re: Produnction : Flink 1.14.4 : Kafka reader threads blocked

Hi Vignesh, I think you can check the following things: 1. Check the cpu usage of the workers. Are they close to zero or almost full? 2. Any back pressure happened in downstream tasks? 3. Is the fullGC significant serious? Best, Yuan > On 7 Apr 2022, at 12:33 PM, Vignesh Ramesh wrote: > > H

Re: flink pipeline handles very small amount of messages in a second (only 500)

Hi Sigalit, In your settings, I guess each job will only have one slot (parallelism). So is it too many input for your jobs with parallelism only one? One easy way to confirm is that you increase your slots and job parallelism twice and then see whether the QPS is increased. Hope this would

Re: how to achieve sideOutputLateData() in FlinkSQL?

Hi, in my understanding, Flink only support to get late data by side output in data stream api currently. For Table API/SQL, unfortunately, late events will always be dropped. You can see this link as reference: https://stackoverflow.com/questions/60218235/using-event-time-with-lateness-in-flin

Re: onTimer() of a KeyedProcessFunction stops getting triggered after a while

After fixing your negative timestamp bug, can the timer be triggered? > On 23 Mar 2022, at 2:39 AM, Binil Benjamin wrote: > > Here are some more findings as I was debugging this. I peeked into the > snapshot to see the current values in "_timer_state/processing_user-timers" > and here is ho

Re: Flink Kinesis checkpoint failures long running service

ient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:903) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1097) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) > at > com.amazonaw

Re: onTimer() of a KeyedProcessFunction stops getting triggered after a while

Hi, can you share your code so we can check whether it is written correctly. > On 18 Mar 2022, at 7:54 AM, Binil Benjamin wrote: > > Hi, > > We have a class that extends KeyedProcessFunction and overrides onTimer() > method. During processElement(), we register a timer callback using > cont

Re: Flink Kinesis checkpoint failures long running service

Hi, so the problem is about checkpoints. We need to understand why there are checkpoint failure. Can you provide more logs. We need to check the log to see more details about the first failed checkpoint. > On 17 Mar 2022, at 9:41 AM, Vijayendra Yadav wrote: > >  > Hi Flink Team, > > I am usi

Re: Questions regarding connecting local FlinkSQL client to remote JobManager in K8s

rvice: jobmanager.rpc.address: rest.port: More info about kubernetes service can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types Hopes my reply help you. > On 3 Mar 2022, at 6:00 PM, yu'an huang wrote: > > Hi Elkhan, >

Re: Questions regarding connecting local FlinkSQL client to remote JobManager in K8s

Hi Elkhan, Except for JM have an external IP address, I think the port 6123 also need to be opened. You may need to set a host port for 6123 in JM pod or expose this port by Kubernetes service. But I am not sure whether the sql-client communicate with JM via Rest endpoint or RPC port. Hopes som

Re: Netty Client Thread - Classloader leak

I suddenly realised that I replied directly to Sudharsan. FYI if anyone wants to check this email too: > On 2 Mar 2022, at 11:36 PM, yu'an huang wrote: > > Hi Sudharsan, > > I think you are right. I just tried your scenario. I set a standalone session > cluster wit