[
https://issues.apache.org/jira/browse/FLINK-36569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17926765#comment-17926765
]
Hongshun Wang commented on FLINK-36569:
---------------------------------------
[~ft20082]
It seems that :
1. if the KafkaWriter is chained with KafkaCommitter, then KafkaWriter will
pass FlinkKafkaInternalProducer to KafkaCommitter(without serialize and
deserialize). In this case,
KafkaWriter and KafkaCommitter share one FlinkKafkaInternalProducer. Thus,
recyclable.ifPresent(Recyclable::close) will valid. (However, if the commit
lags too much, there are also more and more FlinkKafkaInternalProducer.
2. f the KafkaWriter is chained with KafkaCommitter, then KafkaWriter and
KafkaCommitter will each have a different FlinkKafkaInternalProducer object
with same producerId. In this case, the FlinkKafkaInternalProducer of
KafkaWriter will never be recycled.
> flink kafka connector do not close kafka produer when it checkpoint success
> ---------------------------------------------------------------------------
>
> Key: FLINK-36569
> URL: https://issues.apache.org/jira/browse/FLINK-36569
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 1.19.0, 1.20.0
> Environment: flink: 1.20
> flink kafka connector: 3.3.0-1.20
> Reporter: Jake.zhang
> Priority: Major
> Attachments: image-2024-10-18-13-31-39-253.png,
> image-2024-10-21-14-02-41-823.png
>
>
> flink kafka connector do't close FlinkKafkaInternalProducer when flink
> checkpoint success in flink 1.20/1.19 . it will create one
> FlinkKafkaInternalProducer per checkpoint.
>
> FlinkKafkaInternalProducer do not close automatic. so kafka producer network
> thread will more and more
> it create `getRecoveryProducer` each time, `recyclable` object always
> null, so `recyclable.ifPresent(Recyclable::close)` not work.
> `org.apache.flink.connector.kafka.sink.KafkaCommitter`
> {code:java}
> producer =
> recyclable
> .<FlinkKafkaInternalProducer<?, ?>>map(Recyclable::getObject)
> .orElseGet(() -> getRecoveryProducer(committable));
> producer.commitTransaction();
> producer.flush();
> recyclable.ifPresent(Recyclable::close);{code}
>
> !image-2024-10-21-14-02-41-823.png!
>
> !image-2024-10-18-13-31-39-253.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
