Re: Network Buffers

Hi, Pritam. This error message indicates that the current configuration of the network buffer is not enough to handle the current workload. > What is the meaning of this exception (The total number of network buffers > is currently set to 22773 of 32768 bytes each)? > This just provides some

Re: WELCOME to user@flink.apache.org

Hi, Thanks Li Shao, I got it. You can analyze the specific memory usage of metaspace. In fact, we have also encountered the same problem of running batch jobs in the session cluster, which resulted in metaspace growth due to the classloader. I have created a PR [1] for FLIP-32265 [2] for this and

Re: flink14 sql sink kafka error

Hi, 湘晗刚, This error seem to be an error from the Kafka server. Maybe you should check whether the Kafka server occurs some error. Or you could provide more messages about the request. These information is too short to analyze, Best, Hang 湘晗刚 <1016465...@qq.com> 于2023年6月5日周一 15:08写道： >

Re: Custom Counter on Flink File Source

Hi, Kirti Dhar Upadhyay K. We could get the metric group from the context, like `SourceReaderContext` and `SplitEnumeratorContext`. These contexts could be found when creating readers and enumerators. See `AbstractFileSource#createReader` and `AbstractFileSource#createEnumerator`. Best, Hang

回复：python 自定义sink

退订 回复的原邮件 | 发件人 | smq<374060...@qq.com.invalid> | | 发送日期 | 2023年05月30日 12:22 | | 收件人 | user-zh | | 主题 | python 自定义sink | java中可以继承richainkfunction和checkpointedfunction 两个类实现自定义sink。在python中如何实现这种功能呢

Re: WELCOME to user@flink.apache.org

Hi Li Shao, Currently Flink will create a user classloader in JobManager for each job which can only be released by FullGC, I think this is why JVM metaspace is increasing, you can check it. Are you using session mode? I have a small question: Is your job SQL only without UDF or DataStream?

Re: Bulk storage of protobuf records in files

Hi Ryan, What I usually encounter is writing Protobuf format data to systems such as Kafka, and I have never encountered writing to a file yet. Best, Shammon FY On Mon, Jun 5, 2023 at 10:50 PM Martijn Visser wrote: > Hey Ryan, > > I've never encountered a use case for writing Protobuf

Re: WELCOME to user@flink.apache.org

Hi, Recently I noticed my job manager JVM metaspace is keeping increasing for running batch flink jobs. I found similar stackoverflow post: https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, but there is no solution on this. I am

Network Buffers

Hi All, java.io.IOException: Insufficient number of network buffers: required 2, but only 0 available. The total number of network buffers is currently set to 22773 of 32768 bytes each. What is the meaning of this exception (The total number of network buffers is currently set to 22773 of 32768

Custom Counter on Flink File Source

Hi Community, I am trying to add a new counter for number of files collected on Flink File Source. Referring the doc https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/ I understand how to add a new counter on any operator. this.counter =

Re: Bulk storage of protobuf records in files

Hey Ryan, I've never encountered a use case for writing Protobuf encoded files to a filesystem. Best regards, Martijn On Fri, May 26, 2023 at 6:39 PM Ryan Skraba via user wrote: > Hello all! > > I discovered while investigating FLINK-32008[1] that we can write to the > filesystem connector

Re: flink14 batch mode can read.iceberg table but stream mode can not

Hi, This question is better suited for the Iceberg community, since they've built the Flink-Iceberg integration. Best regards, Martijn On Wed, May 31, 2023 at 9:48 AM 湘晗刚 <1016465...@qq.com> wrote: > flink14 batch mode can read iceberg table but stream mode can not ,why? > Thanks in advance >

Re: Using pre-registered schemas with avro-confluent-registry format is not possible

Hi Jannik, Can you share how you've set those properties, because I've been able to use this without any problems. Best regards, Martijn On Thu, Jun 1, 2023 at 2:43 PM Schmeier, Jannik wrote: > Hello Thias, > > > > thank you for your answer. > > > > We've tested registering an existing (byte

Re: Flink Stream processing with Dynamic Conditional Sinks

I've considered a similar question before: Dynamic addition of Sinks / based on some external configuration. The answer I've mostly been given is: this is a bad idea. The checkpoint state that flink uses for job recovery is dependent on the topology of the job, and dynamically adding more sinks

Flink Stream processing with Dynamic Conditional Sinks

Hello, I am trying out flink for one stream processing scenario and was wondering if it can be achieved using Apache Flink. So any pointers regarding how it can be achieved will be of great help. Scenario :- A kafka topic has the input for stream processing, multiple applications lets say A & B

flink14 sql sink kafka error

UnknownServerException :The server experienced an unexpected error when processing the reqiest. Thanks Kobe24

Re: High Start-Delay And Aligned Checkpointing Causing Timeout.

Hi, Pritam. I think the definition works for aligned checkpoint and unaligned checkpoint: "The alignment duration, which is defined as the time between receiving first and the last checkpoint barrier. "