[
https://issues.apache.org/jira/browse/KAFKA-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324857#comment-16324857
]
Ivan Babrou commented on KAFKA-6441:
------------------------------------
Looks like the issue is in Sarama, which only reads one record batch:
* https://github.com/Shopify/sarama/issues/1022
> FetchRequest populates buffer of size MinBytes, even if response is smaller
> ---------------------------------------------------------------------------
>
> Key: KAFKA-6441
> URL: https://issues.apache.org/jira/browse/KAFKA-6441
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.1
> Reporter: Ivan Babrou
>
> We're using Sarama Go client as consumer, but I don't think it's relevant.
> Producer is syslog-ng with Kafka output, I'm not quite sure which log format
> Kafka itself is using, but I can assume 0.11.0.0, because that's what is set
> in topic settings.
> Our FetchRequest has minSize = 16MB, maxSize = 64, maxWait = 500ms. For a
> silly reason, Kafka decides to reply with at least minSize buffer with just
> one 1KB log message. When Sarama was using older consumer API, everything was
> okay. When we upgraded to 0.11.0.0 consumer API, consumer traffic for
> 125Mbit/s topic spiked to 55000Mbit/s on the wire and consumer wasn't even
> able to keep up.
> 1KB message in a 16MB buffer is 1,600,000% overhead.
> I don't think there's any valid reason to do this.
> It's also mildly annoying that there is no tag 0.11.0.1 in git, looking at
> changes is harder than it should be.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
