[
https://issues.apache.org/jira/browse/KAFKA-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751740#comment-13751740
]
Jason Toffaletti commented on KAFKA-1024:
-----------------------------------------
I may have spoken too soon, it might just be taking more time to reach critical
levels. The GC activity reported by jvisualvm is slowly creeping up from rarely
peaking at 0.1% to steadily hovering around 1.3% over the past hour and heap
size has climbed from 50MB to over 80MB.
{noformat}
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 488074 26414648 char[]
2: 15191 20018704 byte[]
3: 486709 11681016 java.lang.String
4: 63839 8693080 * MethodKlass
5: 63839 8252816 * ConstMethodKlass
6: 93927 7522528 java.util.HashMap$Entry[]
7: 5468 5868952 * ConstantPoolKlass
8: 106269 5714200 java.lang.Object[]
9: 91418 5119408 java.util.HashMap
10: 5468 4572616 * InstanceKlassKlass
11: 4538 3346656 * ConstantPoolCacheKlass
12: 79237 3169480 java.util.TreeMap$Entry
13: 45404 2912840 java.util.Hashtable$Entry[]
14: 90667 2901344 java.util.Vector
15: 45352 2539712 java.util.Hashtable
16: 45330 1813200 java.security.ProtectionDomain
17: 14911 1650304 int[]
...
Total : 2327288 140772592
Heap traversal took 46.218 seconds.
{noformat}
I'll update this in a few hours to get an idea where the difference is. My
messages aren't complex, I'm using DefaultEncoder with KeyedMessage<byte[],
byte[]> and null keys, so I thought I had a pretty good idea what size the
messages are.
I'm running this test on dev machines being used for many other purposes so it
is very likely zookeeper will be slow and I'll end up hitting the max queue
depth during a metadata refresh. However, I've not seen dropped messages on any
producers yet.
> possible memory leak in 0.8 beta1 producer with G1GC
> ----------------------------------------------------
>
> Key: KAFKA-1024
> URL: https://issues.apache.org/jira/browse/KAFKA-1024
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Environment: java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> Reporter: Jason Toffaletti
>
> I have this in my pom.xml
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.9.2</artifactId>
> <version>0.8.0-beta1</version>
> </dependency>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.9.2</version>
> </dependency>
> <dependency>
> <groupId>org.xerial.snappy</groupId>
> <artifactId>snappy-java</artifactId>
> <version>1.0.4.1</version>
> </dependency>
> I'm using snappy compression codec and producing about 7k msg/sec on average.
> I'm producing batches up to 10 messages per batch with null keys to a topic
> with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using
> XX:+UseG1GC. After about 12 hours of operation heap usage hits right up
> against the 64MB limit and the producer drops to about 4k msg/sec because of
> the GC pressure. When I restart the process the heap usage goes back to
> normal (around 30MB) and the producer does 7k msg/sec again.
> What else can I provide to help debug this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
