Hi all, I'd like to start a discussion on KIP-1386, which lets an application set the record limit on each call to poll() rather than only through max.poll.records at construction.
When processing slows down, a batch sized for normal load stops fitting inside max.poll.interval.ms. The consumer is evicted, rejoins, asks for another batch, and is evicted again until the slowdown passes or the consumer is restarted with larger max.poll.interval.ms (or smaller max.poll.records) often with an educated guess because there’s no visibility on how much the consumer missed the deadline by. The KIP adds poll(PollOptions) with an optional maxRecords, bounded by max.poll.records so a call can only ask for less. It also adds consumer metrics for how much of the poll interval is left. It pairs with KIP-1364: once the rebalance cause is exposed, an application evicted for exceeding the poll interval can come back with a smaller batch. Looking forward to your feedback. KIP link: https://cwiki.apache.org/confluence/x/RwKDGw Thanks, Aditya
