[
https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536705#comment-13536705
]
Joel Koshy commented on KAFKA-598:
----------------------------------
33.2: not really, because that would violate the new config's semantics - i.e.,
each thread shouldn't exceed it's allocated amount of memory.
That said, I just realized that this implementation has couple flaws and may
need to be refactored or have its scope reduced.
The max mem config is not always respected in this implementation. When we do
the serial fetches the queue will have larger chunks
(than the fair partition fetch size). However, the function that computes the
fair partition fetch size assumes that the blocking queue only
has chunks of the fair fetch size. I think we can take care of this, but will
think about it a bit more. Another problem is that the aggregate fetch
size is the fair size * num partitions assigned to the thread. So for example
partition assignment happens to be very skewed and a thread
happens to have only one partition; a serial refetch will be pointless since it
can't use a larger fetch size.
> decouple fetch size from max message size
> -----------------------------------------
>
> Key: KAFKA-598
> URL: https://issues.apache.org/jira/browse/KAFKA-598
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Attachments: KAFKA-598-v1.patch, KAFKA-598-v2.patch,
> KAFKA-598-v3.patch
>
>
> Currently, a consumer has to set fetch size larger than the max message size.
> This increases the memory footprint on the consumer, especially when a large
> number of topic/partition is subscribed. By decoupling the fetch size from
> max message size, we can use a smaller fetch size for normal consumption and
> when hitting a large message (hopefully rare), we automatically increase
> fetch size to max message size temporarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
