[
https://issues.apache.org/jira/browse/KAFKA-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139291#comment-18139291
]
이선민 commented on KAFKA-21241:
-----------------------------
[~muralibasani] [~schofielaj] Following up on the October 20 measurements:
those covered only the assignor, so coordinator and broker capacity remain open
questions.
I'd propose starting with a share-group heartbeat microbenchmark, using 1,000
members as a baseline and scaling to the 20–30k-member use case in this ticket.
The first step would measure steady-state processing cost within the
coordinator shard; a follow-up would measure assignment-record counts and
serialized bytes across join/leave and subsequent reassignment.
I can put together an initial benchmark, using the shard-level setup in
https://github.com/apache/kafka/pull/24101 as a reference.
Would this be the most useful next step for the KIP, or would you prefer
broker-side ShareSessionCache memory measurements first, varying active
sessions and cached partitions per session?
If there's already a KIP draft, I'd be happy to align the measurements with its
scope.
> Increase the number of consumers in the share consumer group
> ------------------------------------------------------------
>
> Key: KAFKA-21241
> URL: https://issues.apache.org/jira/browse/KAFKA-21241
> Project: Kafka
> Issue Type: Improvement
> Reporter: Prakhar Kulshrestha
> Priority: Major
> Attachments: MapBench.java, WorkingSetBench.java,
> assignor-gc-G1.json, assignor-gc-Parallel.json, assignor-sweep-pretouch.json,
> hashmap-isolated.json, notes.txt, working-set.json
>
>
> Currently the max number of consumers in a share consumer group is default
> to 200 and max to 1000. In bigger platforms where batch processing along
> with slow consumer use cases limiting to just 1000 consumer is very
> restrictive.{*}{*}
> One of the scenario is a batch processing of millions of records and having
> to parallel process 20-30k records in parallel using spring kafka concurrency
> feature would mean we would need at least 20-30k consumers in a consumer
> groups with around 1k pods. Current 1000 limit just let it have ~60pods.
> Request this to be reevaluated and increase the number of consumers that a
> share group can have.
> *group.share.max.size*
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
