Umesh Batra created KAFKA-1795:
----------------------------------
Summary: OOME - high level kafka consumer
Key: KAFKA-1795
URL: https://issues.apache.org/jira/browse/KAFKA-1795
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.8.1.1, 0.9.0
Reporter: Umesh Batra
Assignee: Neha Narkhede
Priority: Blocker
I am using Kafka High Level Consumer, version kafka_2.8.0 - 0.8.1.1 with
zkclient - 0.3 and I get OOME just after 10/15 minutes, My volume test setup
has just one topic with 10 partitions with continuous message (size ~500KB)
flow and below are my configuration;
# consumer broker properties
#####################################################
zookeeper.connect=localhost:2181,localhost:2181
group.id=tc
consumer.id=tc
zookeeper.sync.time.ms=200
zookeeper.connection.timeout.ms=10000
zookeeper.session.timeout.ms=10000
fetch.size=500000
fetch.message.max.bytes=1000000
auto.commit.enable=true
auto.commit.interval.ms=100
auto.offset.reset=largest
queued.max.message.chunks=1
backoff.increment.ms=1000
rebalance.max.retries=10
rebalance.retries.max=10
rebalance.backoff.ms=10000
refresh.leader.backoff.ms=20000
consumer.timeout.ms=50000
Memory histogram shows 60% of memory consumed by byte[] and most of the
remaining by char[] and HashMap$Node. In my various tries to recover from the
situation. I observed "metric-*" thread live even after I shutdown Kafka
connector?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
