Shantanu Deshmukh created KAFKA-7422:
----------------------------------------
Summary: Huge memory consumption (leak?) by Kafka Producer
Key: KAFKA-7422
URL: https://issues.apache.org/jira/browse/KAFKA-7422
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.10.1.0
Environment: RedHat Linux 7.3
Reporter: Shantanu Deshmukh
Attachments: heapdump.20180918.111423.4587876.0002.phd
Hello,
We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some
applications which have Kafka Producers embedded in them which send application
logs to a topic. This topic has 10 partitions with replication factor of 3.
We are observing that memory usage on some of these application servers keep
shooting through the roof intermittently. After taking heapdump we found out
that top suspects were:
*---------------------*
*org.apache.kafka.common.network.Selector -*
occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in one
instance of *"byte[]"* loaded by *"<system class loader>"*.
*org.apache.kafka.common.network.KafkaChannel -*
occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in one
instance of *"byte[]"* loaded by *"<system class loader>"*
*---------------------*
Both of these were holding about 352MB of space. 3 such instances, so they were
consuming about 1.2GB of memory.
Now regarding usage of producers. Not a huge amount of logs are being sent to
Kafka cluster. It is about 200 msgs/sec. Only one producer object is being used
throughout application. Async send function is used.
What could be the cause of such huge memory usage? Is this some sort of memory
leak in this specific Kafka version?
Here's producer config being used at the present.
{{>>>>>>>>>>>>>>>>}}
{{kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092}}
{{ kafka.acks=0}}
{{ kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer}}
{{
kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer}}
{{[kafka.max.block.ms|http://kafka.max.block.ms/]=1000}}
{{[kafka.request.timeout.ms|http://kafka.request.timeout.ms/]=1000}}
{{ kafka.max.in.flight.requests.per.connection=1}}
{{ kafka.retries=0}}
{{ kafka.compression.type=gzip}}
{{ kafka.security.protocol=SSL}}
{{ kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks}}
{{ kafka.ssl.truststore.password=XXXXXX}}
{{ logger.level=INFO }}{{<<<<<<<<<<<<<<}}
Attaching heapdump for your perusal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
