Re: Flink Job Failed With Kafka Exception

2023-11-05 Thread Hang Ruan
Hi, Madan. This error seems like that there are some problems when the consumer tries to read the topic metadata. If you use the same source for these topics, the kafka connector cannot skip one of them. As you say, you need to modify the connector's default behavior. Maybe you should read the

Re: Flink Job Failed With Kafka Exception

2023-11-05 Thread Junrui Lee
Hi Madan, Do you mean you want to restart only the failed tasks, rather than restarting the entire pipeline region? As far as I know, currently Flink does not support task-level restart, but requires restarting the pipeline region. Best, Junrui Madan D via user 于2023年10月11日周三 12:37写道: > Hello

Re: Instructions / steps regarding contributing : flink-training

2023-11-05 Thread Junrui Lee
Hello Milind, Thank you for your interest in contributing to flink-training! To add or modify unit tests, please start by creating a JIRA ticket at the FLINK JIRA page: https://issues.apache.org/jira/browse/FLINK, and set the "Component/s" to "Documentation / Training / Exercises" for your

Re: How to tell if job is being restarted in log?

2023-11-05 Thread Junrui Lee
Hi John, If you want to know more details about why your job is restarting, you can search for the keyword "to FAILED" in the JobManager logs. These log entries will show you the timing of each restart and the associated exception information. Additionally, you can check the exception page to

Re: Inquiry about ActiveResourceManager and StandaloneResourceManager in Flink

2023-11-05 Thread Junrui Lee
Hi, Steven Regarding the term "active" in ActiveResourceManager, it signifies that it is responsible for actively creating and managing TaskManager instances based on requirements. It dynamically adjusts resources in the cluster, starting new TaskManager instances when needed. On the other hand,

Re: Checkpoints are not triggering when S3 is unavailable

2023-11-05 Thread Hangxiang Yu
Hi, Do you mean your checkpoint failure stops the normal running of your job? What's your sink type? If it relies on the completed checkpoint to commit, it should be expected. On Tue, Oct 31, 2023 at 12:03 AM Evgeniy Lyutikov wrote: > Hi team! > I came across strange behavior in Flink 1.17.1.

Re: Clear the State Backends in Flink

2023-11-05 Thread Hangxiang Yu
Hi, Arjun. Do you mean clearing all states stored in a user-defined state ? IIUC, It could be done for Operator state. But it cannot be done for Keyed state for users because every operation for it is binded with a specific key currently. BTW, Could you also share your business scenario ? It could