Jun Rao created KAFKA-21397:
-------------------------------
Summary: a single large batch bigger than the quota's allowed
burst can stall consumers indefinitely
Key: KAFKA-21397
URL: https://issues.apache.org/jira/browse/KAFKA-21397
Project: Kafka
Issue Type: Bug
Components: core
Reporter: Jun Rao
How the stall happens: The fetch size is capped to fit the quota. For consumer
fetches, KafkaApis.scala:740-748 caps fetchMaxBytes at maxValueInQuotaWindow.
That is quota × (quota.window.num − 1) × quota.window.size.seconds
(ClientQuotaManager.java:377-383), so about 10× the per-second quota with the
defaults.
The first batch ignores the cap. ReplicaManager.readFromLog reads the first
non-empty partition with minOneMessage = true (ReplicaManager.scala:1896-1906).
KafkaApis.scala:704-721 records the response size. That alone puts the client
over its quota, so the throttle time is greater than 0. The broker then removes
the bytes it just recorded and returns an empty throttled response.
The consumer retries from the same offset after the throttle time. The quota is
still configured based on the configured window sizes and doesn't change. The
consumer gets the same empty response.
Share fetch has the same pattern (KafkaApis.scala:3559-3561 and :4327).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
