[
https://issues.apache.org/jira/browse/FLINK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-33239:
-----------------------------------
Priority: Major (was: Blocker)
> After enabling exactly-once in the Flink Kafka sink, the Kafka broker's
> memory keeps increasing, eventually causing the Kafka broker to crash.
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-33239
> URL: https://issues.apache.org/jira/browse/FLINK-33239
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.17.1
> Environment: flink 1.17.1
> kafka server 2.8.2
> Reporter: Darcy Lin
> Priority: Major
> Attachments: image-2023-10-11-18-47-32-712.png
>
>
> We are using Flink version 1.17.1 and Kafka server version 2.8.2. After
> enabling exactly-once, in order to allow downstream consumers to read data
> from Kafka as soon as possible, we set the checkpoint interval to 5 seconds.
> Approximately three days after writing to the Kafka cluster, the Kafka JVM's
> memory is exhausted. We printed the memory consumption and found that the
> main consumption is on the {{kafka.log.ProducerStateEntry}} object.
> Currently, in the exactly-once Kafka sink, a new producer is created every
> time a checkpoint is executed. The {{kafka.log.ProducerStateEntry}} object
> seems to store the producer's state, so it keeps increasing. We'd like to
> ask: Is this normal? If it's normal, do we need to allocate a large amount of
> memory for our Kafka cluster? If it's not normal, how should we solve this
> problem?
> !image-2023-10-11-18-47-32-712.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
