[ 
https://issues.apache.org/jira/browse/KAFKA-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124752#comment-18124752
 ] 

Muralidhar Basani commented on KAFKA-21241:
-------------------------------------------

[~prakhar1801] thanks for the report.

I'm not sure I follow why this workload needs 20–30k separate consumers. Share 
groups don't tie consumer count to partition count, so parallelism doesn't have 
to come from the number of consumer instances.

Because KafkaShareConsumer is not thread safe, did you try fewer consumers per 
pod and acking them from consumer thread once it is complete? and splitting the 
load across several share groups on the same topic.

Changing this limit group.share.max.size means changing several other dependent 
configs like group.share.max.share.sessions, rate limits, running bench marks 
and KIP etc.

(There is a KIP-1261 touching share group configs, but not this one in 
particular.)

> Increase the number of consumers in the share consumer group
> ------------------------------------------------------------
>
>                 Key: KAFKA-21241
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21241
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Prakhar Kulshrestha
>            Priority: Major
>
> Currently the max  number of consumers in a share consumer group is default 
> to 200 and max to 1000.  In bigger platforms where batch processing along 
> with slow consumer use cases limiting to just 1000 consumer is very 
> restrictive.{*}{*}
> One of the scenario is a batch processing of millions of records and having 
> to parallel process 20-30k records in parallel using spring kafka concurrency 
> feature would mean we would need at least 20-30k consumers in a consumer 
> groups with around 1k pods. Current 1000 limit just let it have ~60pods.
> Request this to be reevaluated and increase the number of consumers that a 
> share group can have.
> *group.share.max.size*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to