Re: Backpressure handling in FileSource APIs - Flink 1.16

Thanks Shammon for clarification. On Wed, May 24, 2023 at 11:01 AM Shammon FY wrote: > Hi Kamal, > > The source will slow down when there is backpressure in the flink job, you > can refer to docs [1] and [2] to get more detailed information about > backpressure mechanism. > > Currently there's n

Re: Backpressure handling in FileSource APIs - Flink 1.16

Hi Kamal, The source will slow down when there is backpressure in the flink job, you can refer to docs [1] and [2] to get more detailed information about backpressure mechanism. Currently there's no API or Callback in source for users to do some customized operations for backpressure, but users c

RE: ClassNotFoundException when deploying to k8s using linkerd

Replying to myself with some delay admittedly ;-) The issue had nothing to do with Linkerd, but was basically a class loading issue due to a missing env var in the taskmanager. In case you are interested in the details, and some clarifications on how Linkerd needs to be setup with Flink, see my d

Reading KafkaSource state from a savepoint using the State Processor API

Hi everyone, I have a few questions about reading KafkaSource state using the State Processor API. I have a simple Flink application which reads from a Kafka topic then produces to a different topic. After running the Flink job and stopping it with a savepoint, I then write a few more records to t

Re: python udf with flinksql

Thanks Dian, that resolved my issues. On Sun, May 21, 2023 at 6:55 PM Dian Fu wrote: > > Hi Tom, > > The following statement is incorrect. > ``` > CREATE FUNCTION add AS 'custom_udf_2.MyUDF' > LANGUAGE PYTHON; > ``` > > You should define it as following: > custom_udf_2.py > ``` > from pyflink.tab

Kafka Quotas & Consumer Group Client ID (Flink 1.15)

Hello, I have two questions that are related to each other: *First question:* I have been trying to set `client.id` to set a kafka client quota for consumer_byte_rate since whenever our kafka job gets redeployed it reads a lot of data f

[no subject]

Hello! I have deployed the flink kubernetes operator 1.4.0 on minikube and I enabled the autoscaler. However, I get this error "2023-05-22 14:54:25,494 o.a.f.k.o.a.m.ScalingMetrics [ERROR][default/example] Cannot compute true processing/output rate without busyTimeMsPerSecond " on logs of the ope

[no subject]

Hello! I have deployed the flink kubernetes operator 1.4.0 on minikube and I enabled the autoscaler. However, I get this error "2023-05-22 14:54:25,494 o.a.f.k.o.a.m.ScalingMetrics [ERROR][default/example] Cannot compute true processing/output rate without busyTimeMsPerSecond " on logs of the ope

Re: Backpressure handling in FileSource APIs - Flink 1.16

Hello Community, Can you please share views about the query asked above w.r.t back pressure for FileSource APIs for Bulk and Record stream formats. Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion. Rgds, Kamal On Tue, 23 May 2023, 12:26 pm Kamal Mittal, wrote: > Adde

Re: In HA mode, support the same application run multiple jobs?

Hi High-Availability in Application Mode is only supported for single-execute() applications.[1] And the reason is[2]: The added complexity stems mainly from the fact that the different jobs > within an application may at any point be in different stages of their > execution, e.g. some may be r

Re: Flink Kubernetes Operator lifecycle state count metrics question

Hi Andrew! I think you are completely right, this is a bug. The per namespace metrics do not seem to filter per namespace and show the aggregated global count for each namespace: I opened a ticket: https://issues.apache.org/jira/browse/FLINK-32164 Thanks for reporting this! Gyula On Mon, May 22

Re: Question about Flink exception handling

Thanks for the clarification. On Tue, May 23, 2023 at 7:07 PM Weihua Hu wrote: > Hi Sharif, > > You could not catch exceptions globally. > > For exceptions that can be explicitly ignored for your business, you need > to add a try-catch in the operators. > For exceptions that are not catched, Fli

Re: Question about Flink exception handling

Hi Sharif, You could not catch exceptions globally. For exceptions that can be explicitly ignored for your business, you need to add a try-catch in the operators. For exceptions that are not catched, Flink will trigger a recovery from failure automatically[1]. [1] https://nightlies.apache.org/fl

Re: Flink SQL Async UDF

Hi Giannis: I think this is due to the User Defined AsyncTableFunction have not been supported yet. It has a little different in type inference. I just opened a thread discuss about supporting this feature, you can refer to: https://lists.apache.org/thread/7vk1799ryvrz4lsm5254q64ctm89mx2l Than

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.5.0 released

Niceee. Thanks for managing the release, Gyula! -Max On Wed, May 17, 2023 at 8:25 PM Márton Balassi wrote: > > Thanks, awesome! :-) > > On Wed, May 17, 2023 at 2:24 PM Gyula Fóra wrote: >> >> The Apache Flink community is very happy to announce the release of Apache >> Flink Kubernetes Operato

Re: IRSA with Flink S3a connector

Hi Anuj, I recalled another ticket on this topic, which had some things to test. I don't know if that resolved the issue, can you verify it? See https://issues.apache.org/jira/browse/FLINK-31095 Best regards, Martijn On Tue, May 23, 2023 at 7:04 AM Anuj Jain wrote: > Hello, > Please provide s

In HA mode, support the same application run multiple jobs?

In HA mode, support the same application run multiple jobs?