Hi Daniel, Thanks for reaching out, we are constantly trying to improve the reliability of our connectors. I assume you are running the KafkaSink with exactly-once delivery guarantee. On startup, the KafkaSink tries to abort lingering transactions from previous executions. Unfortunately, nothing comes to my mind immediately why your job hangs.
Can you maybe share the logs with us from such a run? It would be also great to know more information about your environment e.g. bounded or unbounded jobs, parallelism, Kafka client/server version, potential topic acls, transactional id prefix. Best, Fabian On Mon, Jan 31, 2022 at 3:27 PM Daniel Peled <[email protected]> wrote: > > Hi everyone, > > Has anyone encountered any problem with the new KafkaSink that is used in > Flink 1.14 ? > > When running our jobs, the sinks of some of our jobs are stuck in > initializing for more than an hour. > The only thing that helps is deleting the topic __transaction_state. > After deleting this topic, all sinks are immediately released and are in > running status. > The problem is quite random each time in a different job. > There are times that all jobs start running without any problems. > > Unfortunately we had to go back to the deprecated FlinkKafkaProducer. > > We didn't have these problems with Flink 1.13 and FlinkKafkaProducer > > Any ideas on what to do ? > What are we doing wrong? > > BR, > Daniel >
