[
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863201#comment-13863201
]
Neha Narkhede commented on KAFKA-1196:
--------------------------------------
As suggested by Jun -
If a broker is the leader of multiple partitions of a topic, the high level
consumer will fetch all those partitions in a single fetch request. Then
the aggregate of the fetched data from multiple partitions could be more
than 2GB.
You can try using more consumers in the same consumer group to reduce #
partitions fetched per consumer.
Thanks,
Jun
> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---------------------------------------------------------------------------
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala
> 2.9.2
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Neha Narkhede
> Priority: Blocker
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if
> compression is used or not.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
