[
https://issues.apache.org/jira/browse/FLINK-31914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
datariver updated FLINK-31914:
------------------------------
Description:
Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing
will be used. KafkaWriter will create FlinkKafkaInternalProducer in the
initialization and snapshotState methods, but there is no place to close it. As
Checkpoints increase, Producers will continue to accumulate. Each Producer
maintains a Buffer, which will cause memory leaks and Job OOM.
By dumping an in-memory instance of Task Manager, you can see that there are a
lot of Producers:
!image-2023-04-25-13-47-25-703.png!
was:
Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing
will be used. KafkaWriter will create FlinkKafkaInternalProducer in the
initialization and snapshotState methods, but there is no place to close it. As
Checkpoints increase, Producers will continue to accumulate. Each Producer
maintains a Buffer, which will cause memory leaks and Job OOM.
By dumping an in-memory instance of Task Manager, you can see that there are a
lot of Producers:
!image-2023-04-24-16-11-22-251.png!
> Failing to close FlinkKafkaInternalProducer created in KafkaWriter with
> exactly-once semantic results in memory leak
> --------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-31914
> URL: https://issues.apache.org/jira/browse/FLINK-31914
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.15.0, 1.16.0, 1.17.0
> Reporter: datariver
> Priority: Major
> Attachments: image-2023-04-25-13-47-25-703.png
>
>
> Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional
> writing will be used. KafkaWriter will create FlinkKafkaInternalProducer in
> the initialization and snapshotState methods, but there is no place to close
> it. As Checkpoints increase, Producers will continue to accumulate. Each
> Producer maintains a Buffer, which will cause memory leaks and Job OOM.
> By dumping an in-memory instance of Task Manager, you can see that there are
> a lot of Producers:
> !image-2023-04-25-13-47-25-703.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
