[
https://issues.apache.org/jira/browse/KAFKA-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124773#comment-18124773
]
Prakhar Kulshrestha commented on KAFKA-21241:
---------------------------------------------
Thanks for your response [~muralibasani] !
The problem with fewer consumer per pod is if I fetch say 5 messages per
consumer and I have to ack them, I have to ack for all of them together which
means if 4 messages takes 5secs and 1 message takes 1min all the other 4
threads are just waiting for the longest running one to complete. This impacts
the overall throughput of the app.
re "and splitting the load across several share groups on the same topic"-- Can
you explain bit more on this. My understanding is if I have more than one share
group on a topic, consumers of the respective groups would get all the messages
similar to how consumer groups work. So if I have one topic with 10 messages
and 2 share groups with 1 consumer each, both share group consumers would get
all the 10 messages.
I was discussing this with [~schofielaj] on the slack and was hoping to have a
KIP to reevaluate this config for 4.5 or later.
> Increase the number of consumers in the share consumer group
> ------------------------------------------------------------
>
> Key: KAFKA-21241
> URL: https://issues.apache.org/jira/browse/KAFKA-21241
> Project: Kafka
> Issue Type: Improvement
> Reporter: Prakhar Kulshrestha
> Priority: Major
>
> Currently the max number of consumers in a share consumer group is default
> to 200 and max to 1000. In bigger platforms where batch processing along
> with slow consumer use cases limiting to just 1000 consumer is very
> restrictive.{*}{*}
> One of the scenario is a batch processing of millions of records and having
> to parallel process 20-30k records in parallel using spring kafka concurrency
> feature would mean we would need at least 20-30k consumers in a consumer
> groups with around 1k pods. Current 1000 limit just let it have ~60pods.
> Request this to be reevaluated and increase the number of consumers that a
> share group can have.
> *group.share.max.size*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)