[
https://issues.apache.org/jira/browse/KAFKA-10518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Lee updated KAFKA-10518:
----------------------------
Attachment: kafka-slow-consumer-repro.tar.gz
> Consumer fetches could be inefficient when lags are unbalanced
> --------------------------------------------------------------
>
> Key: KAFKA-10518
> URL: https://issues.apache.org/jira/browse/KAFKA-10518
> Project: Kafka
> Issue Type: Bug
> Reporter: Dhruvil Shah
> Priority: Major
> Attachments: kafka-slow-consumer-repro.tar.gz
>
>
> Consumer fetches are inefficient when lags are imbalanced across partitions,
> due to head of the line blocking and the behavior of blocking for
> `max.wait.ms` until data is available.
> When the consumer receives a fetch response, it prepares the next fetch
> request and sends it out. The caveat is that the subsequent fetch request
> would explicitly exclude partitions for which the consumer received data in
> the previous round. This is to allow the consumer application to drain the
> data for those partitions, until the consumer fetches the other partitions it
> is subscribed to.
> This behavior does not play out too well if the consumer is consuming when
> the lag is unbalanced, because it would receive data for the partitions it is
> lagging on, and then it would send a fetch request for partitions that do not
> have any data (or have little data). The latter will end up blocking for
> fetch.max.wait.ms on the broker before an empty response is sent back. This
> slows down the consumer’s overall consumption throughput since
> fetch.max.wait.ms is 500ms by default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
