Darcy Lin created FLINK-33239:
---------------------------------
Summary: After enabling exactly-once in the Flink Kafka sink, the
Kafka broker's memory keeps increasing, eventually causing the Kafka broker to
crash.
Key: FLINK-33239
URL: https://issues.apache.org/jira/browse/FLINK-33239
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.17.1
Environment: flink 1.17.1
kafka server 2.8.2
Reporter: Darcy Lin
Attachments: image-2023-10-11-18-47-32-712.png
We are using Flink version 1.17.1 and Kafka server version 2.8.2. After
enabling exactly-once, in order to allow downstream consumers to read data from
Kafka as soon as possible, we set the checkpoint interval to 5 seconds.
Approximately three days after writing to the Kafka cluster, the Kafka JVM's
memory is exhausted. We printed the memory consumption and found that the main
consumption is on the {{kafka.log.ProducerStateEntry}} object.
Currently, in the exactly-once Kafka sink, a new producer is created every time
a checkpoint is executed. The {{kafka.log.ProducerStateEntry}} object seems to
store the producer's state, so it keeps increasing. We'd like to ask: Is this
normal? If it's normal, do we need to allocate a large amount of memory for our
Kafka cluster? If it's not normal, how should we solve this problem?
!image-2023-10-11-18-47-32-712.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
