[
https://issues.apache.org/jira/browse/KAFKA-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kousik updated KAFKA-21350:
----------------------------------
Component/s: clients
consumer
Labels: kip (was: )
> Per-call record limit for Consumer poll
> ---------------------------------------
>
> Key: KAFKA-21350
> URL: https://issues.apache.org/jira/browse/KAFKA-21350
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Aditya Kousik
> Assignee: Aditya Kousik
> Priority: Major
> Labels: kip
>
> {{max.poll.records}} and {{max.poll.interval.ms}} are read when a consumer is
> constructed and neither can be changed afterwards.
> If processing slows down, a batch that used to fit inside the poll interval
> stops fitting. The consumer misses its deadline and leaves the group. The
> assignment is released through {{{}onPartitionsLost{}}}, so the in-flight
> batch was never committed and the next owner reprocesses it.
> The consumer rejoins and asks for the same batch size without any changes to
> config. It misses the deadline again. This continues until the slowdown ends.
> A shared dependency degrades for all members at once, so they hit it together
> and the group rebalances repeatedly. The common response is a process restart
> after increasing {{max.poll.interval.ms}} or decreasing
> {{{}max.poll.records{}}}, which reassigns every group in that JVM.
> We can consider providing an API to lower the records from each poll
> operation to fit the best batch size that the client can currently deliver
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
