[
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863212#comment-13863212
]
Gerrit Jansen van Vuuren commented on KAFKA-1196:
-------------------------------------------------
Hi, thanks for the response, but this is not really a solution.
Its obvious that how the api does the call is then flawed that or the broker
should take care of it. I cannot add more consumers just for this issue because
my data will grow maybe it doubles in a year, then I'd run again into the same
problem, which is in the end an integer overflow problem and not a resource
problem.
i.e. beyond a certain amount of data the api fails and I should add more
consumers? I've opted for writing my own api that does calls on a
topic,partition basis to each broker, so far its working.
> java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
> ---------------------------------------------------------------------------
>
> Key: KAFKA-1196
> URL: https://issues.apache.org/jira/browse/KAFKA-1196
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Environment: running java 1.7, linux and kafka compiled against scala
> 2.9.2
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Neha Narkhede
> Priority: Blocker
> Fix For: 0.9.0
>
>
> I have 6 topics each with 8 partitions spread over 4 kafka servers.
> the servers are 24 core 72 gig ram.
> While consuming from the topics I get an IlegalArgumentException and all
> consumption stops, the error keeps on throwing.
> I've tracked it down to FectchResponse.scala line 33
> The error happens when the FetchResponsePartitionData object's readFrom
> method calls:
> messageSetBuffer.limit(messageSetSize)
> I put in some debug code the the messageSetSize is 671758648, while the
> buffer.capacity() gives 155733313, for some reason the buffer is smaller than
> the required message size.
> I don't know the consumer code enough to debug this. It doesn't matter if
> compression is used or not.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
