[
https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536167#comment-13536167
]
Jun Rao commented on KAFKA-598:
-------------------------------
Thanks for patch v3. Some comments.
30. AbstractFetcherThread: The input parameter partitionFetchSize is
mis-leading. The passed-in value is not really the fetch size per partition.
Instead, this is the max memory size allocated to this fetcher thread (which
handles multiple partitions). So, when calculating the partition fetch size, we
need to divide this number by the # of partitions in this fetcher thread.
31. ConsumerFetcherThread: The following logic seems to be handling overflow
when converting from long to int. This is a bit hard to understand. Could we do
the check explicitly (ie. if longVal > Int.MaxValue set requested to
Int.MaxValue)?
val requested = (config.maxMemory / (partitionCount * (1/* current
fetch */ + config.maxQueuedChunks)).max(1))
.toInt
if (requested < 0)
Int.MaxValue
32. AbstractFetcherManager: Do we really need to maintain partitionCount? Scala
doc didn't say HashMap.size() is linear to # of entries. If this is just a
limitation in scala 2.8. I suggest that we just pay the overhead of traversing
the map for now. Typically, the # of fetcher threads is small.
> decouple fetch size from max message size
> -----------------------------------------
>
> Key: KAFKA-598
> URL: https://issues.apache.org/jira/browse/KAFKA-598
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Attachments: KAFKA-598-v1.patch, KAFKA-598-v2.patch,
> KAFKA-598-v3.patch
>
>
> Currently, a consumer has to set fetch size larger than the max message size.
> This increases the memory footprint on the consumer, especially when a large
> number of topic/partition is subscribed. By decoupling the fetch size from
> max message size, we can use a smaller fetch size for normal consumption and
> when hitting a large message (hopefully rare), we automatically increase
> fetch size to max message size temporarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
