[
https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510696#comment-13510696
]
Jun Rao commented on KAFKA-598:
-------------------------------
Thanks for patch v2. Looks good overhead. A couple of comments.
1. There is one subtle issue. In ConsumerIterator, we throw a
MessageSizeTooLargeException if validBytes is 0. This is sort of our way to
inform the consumer client that the consumption is stuck. With this patch, if
we get a really large message, we will just keep issuing fetch requests in
AbstractFetcherThread and never return the fetched data to the consumer queue.
So the consumer client won't get the MessageSizeTooLargeException. Not sure
what's the best way to solve this. One way is that on retry (with a larger
fetch size), we will pass the data to processPartitionData even if validBytes
is 0.
2. It's possible that when validBytes is 0, we are only fetching 1 partition.
So, retry won't help. But the current patch probably still works. So this may
not be an issue.
3. I would make the default FetchSize 100MB and the default MaxQueuedChunks 1.
> decouple fetch size from max message size
> -----------------------------------------
>
> Key: KAFKA-598
> URL: https://issues.apache.org/jira/browse/KAFKA-598
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Attachments: KAFKA-598-v1.patch, KAFKA-598-v2.patch
>
>
> Currently, a consumer has to set fetch size larger than the max message size.
> This increases the memory footprint on the consumer, especially when a large
> number of topic/partition is subscribed. By decoupling the fetch size from
> max message size, we can use a smaller fetch size for normal consumption and
> when hitting a large message (hopefully rare), we automatically increase
> fetch size to max message size temporarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
