Aditya Kousik created KAFKA-21350:
-------------------------------------
Summary: Per-call record limit for Consumer poll
Key: KAFKA-21350
URL: https://issues.apache.org/jira/browse/KAFKA-21350
Project: Kafka
Issue Type: Improvement
Reporter: Aditya Kousik
Assignee: Aditya Kousik
{{max.poll.records}} and {{max.poll.interval.ms}} are read when a consumer is
constructed and neither can be changed afterwards.
If processing slows down, a batch that used to fit inside the poll interval
stops fitting. The consumer misses its deadline and leaves the group. The
assignment is released through {{{}onPartitionsLost{}}}, so the in-flight batch
was never committed and the next owner reprocesses it.
The consumer rejoins and asks for the same batch size without any changes to
config. It misses the deadline again. This continues until the slowdown ends. A
shared dependency degrades for all members at once, so they hit it together and
the group rebalances repeatedly. The common response is a process restart after
increasing {{max.poll.interval.ms}} or decreasing {{{}max.poll.records{}}},
which reassigns every group in that JVM.
We can consider providing an API to lower the records from each poll operation
to fit the best batch size that the client can currently deliver
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
