Vamsi Subhash Achanta created KAFKA-2134:
--------------------------------------------
Summary: Producer blocked on metric publish
Key: KAFKA-2134
URL: https://issues.apache.org/jira/browse/KAFKA-2134
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.1
Environment: debian7, java8
Reporter: Vamsi Subhash Achanta
Assignee: Jun Rao
Hi,
We have a REST api to publish to a topic. Yesterday, we started noticing that
the producer is not able to produce messages at a good rate and the CLOSE_WAITs
of our producer REST app are very high. All the producer REST requests are
hence timing out.
When we took the thread dump and analysed it, we noticed that the threads are
getting blocked on JmxReporter metricChange. Here is the attached stack trace.
"dw-70 - POST /queues/ekl_bigfoot_marvin_production_1/messages" #70 prio=5
os_prio=0 tid=0x00007f043c8bd000 nid=0x54cf waiting for monitor entry
[0x00007f04363c7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
- waiting to lock <0x00000005c1823860> (a java.lang.Object)
at
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:182)
- locked <0x00000007a5e526c8> (a
org.apache.kafka.common.metrics.Metrics)
at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:165)
- locked <0x00000007a5e526e8> (a org.apache.kafka.common.metrics.Sensor)
When I looked at the code of metricChange method, it uses a synchronised block
on an object resource and it seems that it is held by another.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
