[
https://issues.apache.org/jira/browse/FLINK-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461158#comment-16461158
]
Piotr Nowojski commented on FLINK-9287:
---------------------------------------
Yes, indeed there is such bug. The problem boils down to
{{org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#abort}}
being used for both aborting transactions and closing them at the end of the
operator's lifecycle. For EXACTLY_ONCE in both cases we should be closing
{{FlinkKafkaProducer}} but for AT_LEAST_ONCE and NONE we should be closing them
only during operator close - which is not happening.
I'm not sure, but maybe we should split EXACTLY_ONCE and NON-EXACTLY_ONCE
implementations, instead of trying to squeeze in both of them at the same time
into one class.
CC:
[~tzulitai] [~gjy]
> KafkaProducer011 seems to leak threads when not in exactly-once mode
> --------------------------------------------------------------------
>
> Key: FLINK-9287
> URL: https://issues.apache.org/jira/browse/FLINK-9287
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.4.2
> Reporter: Christopher Ng
> Priority: Minor
>
> {{KafkaProducer011}} appears to be leaking {{kafka-producer-network-thread}}
> threads. As far as I can tell it happens when it is not in EXACTLY_ONCE
> mode, it seems that it creates a {{FlinkKafkaProducer}} but never closes it,
> even when the {{FlinkKafkaProducer011}} itself is closed.
> I observed this when running a local cluster and submitting and then
> cancelling a job, a lot of kafka threads were left alive afterwards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
