Re: Limiting backpressure during checkpoints

2022-10-24 Thread Zakelly Lan
Hi Robin, You said that during the checkpoint async phase the CPU is stable at 100%, which is pretty strange to me. Normally the cpu usage of the taskmanager process could exceed 100%, depending on what all the threads are doing. I'm wondering if there is any scheduling mechanism controlling the C

Re: Flink SQL on Docker

2022-10-24 Thread Shengkai Fang
Hi. Could you share the query you use in the tests and let us reproduce this problem offline? It's better you can provide us with more infos: - the Flink version you use - the logs in the sql-client and jm - It's better you can dump the memory to detect which uses the memory Best, Shengkai Trist

Flink SQL on Docker

2022-10-24 Thread Tristan Culp via user
Hello all, I've been working with the Flink SQL Client lately, and have been trying to use it on a Docker image. I've noticed that when it runs on an image, however, it performs much slower and after 1 or 2 sql queries, stops working altogether, leading me to have to restart the SQL client. I've n

Re: Limiting backpressure during checkpoints

2022-10-24 Thread Robin Cassan via user
Hello Yuan Mei! Thanks a lot for your answer :) About the CPU usage, it is pretty stable at 80% normally. Every 15 minutes we trigger a checkpoint, and during this time it is stable at 100% I am starting to wonder if CPU is the real limiting factor, because when checking the Flink UI I see that mo

RE: Skip malformed messages with the KafkaSink

2022-10-24 Thread Salva Alcántara
I finally created this ticket: https://issues.apache.org/jira/browse/FLINK-29480. Can anyone provide any feedback? Regards, Salva On 2022/09/08 10:48:07 Salva Alcántara wrote: > Hi! Is there a way to skip/discard messages when using the KafkaSink, so > that if for some reason messages are malfo

Re: [DISCUSS] Issue tracking workflow

2022-10-24 Thread Xintong Song
I agree with you that option 1) would be the best for us. Let's keep hoping for the best. Option 4), as you said, comes with prices. At the moment, I don't have thorough answers to your questions. Just one quick response, I think there's a good chance that we can import current Jira tickets into

Re: [DISCUSS] Issue tracking workflow

2022-10-24 Thread Martijn Visser
Hi Xintong, I'm also not in favour of option 2, I think that two systems will result in an administrative burden and less-efficient workflow. I'm also not in favour of option 3, I think that this will result in first time users/contributors in not-filling their first bug report, user question or f