[
https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536227#comment-13536227
]
Joel Koshy commented on KAFKA-598:
----------------------------------
Thanks for taking the time to review.
30. I think what I have is correct but may be missing something - can you check
this again and confirm? The passed-in value is (supposed to be) the fetch size
per partition. Only the fetcher manager can compute the fetch size per
partition (since it has access to the partition count for each fetcher thread)
and that function is "wired in" through the constructor of the abstract fetcher
thread.
31. Sure - will make that change.
32. It is true that map size is O(1). However, here I need to take each value
of the map (which is a fetcher thread) and get the number of partitions that it
owns and then sum up the counts. (This is required to compute the partition
fetch size). The iteration over the map values seemed expensive (and it is done
for each fetch request) so I went with the volatile int.
Anyway, I think a better approach would be to maintain a Set of partitions that
the fetcher manager is currently managing and just return the size of the set.
> decouple fetch size from max message size
> -----------------------------------------
>
> Key: KAFKA-598
> URL: https://issues.apache.org/jira/browse/KAFKA-598
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Attachments: KAFKA-598-v1.patch, KAFKA-598-v2.patch,
> KAFKA-598-v3.patch
>
>
> Currently, a consumer has to set fetch size larger than the max message size.
> This increases the memory footprint on the consumer, especially when a large
> number of topic/partition is subscribed. By decoupling the fetch size from
> max message size, we can use a smaller fetch size for normal consumption and
> when hitting a large message (hopefully rare), we automatically increase
> fetch size to max message size temporarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
