[
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869381#comment-13869381
]
Gerrit Jansen van Vuuren commented on KAFKA-1196:
-------------------------------------------------
If I throw an error then what? How can I consume the data in the first place?
I'm using all the defaults and the provided stock consumer/producer.
The problem here is that I've not setup anywhere in the configuration to
consume more than 2GB, its the way that the consumer does its fetch that causes
the data to go over 2GB.
So it means that: If you use the broker and consumer and for some reason the
consumer does a fetch over 2GB at any time, I'll be unable to consume the data
ever, even though no single message is even over the 100-200 mb. The only
solution left to me is then either write my own client, or delete the topic
data every time I see this, which is about 5 seconds after I removed and
recreated the topic.
> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---------------------------------------------------------------------------
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala
> 2.9.2
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: newbie
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if
> compression is used or not.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
