[
https://issues.apache.org/jira/browse/KAFKA-7660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699793#comment-16699793
]
John Roesler commented on KAFKA-7660:
-------------------------------------
> I did not really find the memory leak you mentioned
Huh. I'm looking at the current trunk, and it looks still present. (we add to
the parentSensors map, but never remove from it). I'm not sure how it got
lost... I guess I'll send a quick PR with the fix I had in mind.
> Maybe org.apache.kafka.streams.kstream.internals.metrics.Sensors.java needs
>to use some final String variables here to fix that
[~mjsax] has suggested something similar, at least for the group name, in
[https://github.com/apache/kafka/pull/5795#discussion_r235503158] . I'm
planning to update that PR to save the group name in a constant, but I don't
want to pollute the PR with extra changes for the tags (it's been dragging on
already), so we can add extra constants in a separate PR. (Feel free to send a
PR if you want)
> Does this mean that Sensors/Metrics are only created at streams startup and
>should remain constant after that?
Not quite, the sensors that belong to tasks (and processor nodes, state stores,
etc., which are inside tasks) are created when the task is created and should
remain constant until the task is closed. Tasks get closed when the task gets
migrated to another instance or thread. In this case we *should* remove the
sensors and reclaim the memory. But it sounds from your repro like this is not
happening. Quite possibly, this is a result of the parentSensors map keeping
references to them.
> The memory impact was very small but would probably be more relevant with
> bigger streams topologies.
Yeah, these objects are small, but we'd like people to be able to run Streams
for an arbitrarily long time, over which many task reassignments could occur,
so I still don't like it. Plus memory leaks are gross.
> Stream Metrics - Memory Analysis
> --------------------------------
>
> Key: KAFKA-7660
> URL: https://issues.apache.org/jira/browse/KAFKA-7660
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
> Affects Versions: 2.0.0
> Reporter: Patrik Kleindl
> Priority: Minor
> Attachments: Mem_Collections.jpeg, Mem_DuplicateStrings.jpeg,
> Mem_DuplicateStrings2.jpeg, Mem_Hotspots.jpeg, Mem_KeepAliveSet.jpeg,
> Mem_References.jpeg
>
>
> During the analysis of JVM memory two possible issues were shown which I
> would like to bring to your attention:
> 1) Duplicate strings
> Top findings:
> string_content="stream-processor-node-metrics" count="534,277"
> string_content="processor-node-id" count="148,437"
> string_content="stream-rocksdb-state-metrics" count="41,832"
> string_content="punctuate-latency-avg" count="29,681"
>
> "stream-processor-node-metrics" seems to be used in Sensors.java as a
> literal and not interned.
>
> 2) The HashMap parentSensors from
> org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
> was reported multiple times as suspicious for potentially keeping alive a
> lot of objects. In our case the reported size was 40-50MB each.
> I haven't looked too deep in the code but noticed that the class Sensor.java
> which is used as a key in the HashMap does not implement equals or hashCode
> method. Not sure this is a problem though.
>
> The analysis was done with Dynatrace 7.0
> We are running Confluent 5.0/Kafka2.0-cp1 (Brokers as well as Clients)
>
> Screenshots are attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
