[
https://issues.apache.org/jira/browse/KAFKA-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128995#comment-18128995
]
이선민 commented on KAFKA-21241:
-----------------------------
[^assignor-gc-G1.json] [^assignor-gc-Parallel.json]
[^assignor-sweep-pretouch.json] [^hashmap-isolated.json] [^MapBench.java]
[^notes.txt] [^working-set.json] [^WorkingSetBench.java] [~muralibasani]
[~schofielaj] Following up on the request for benchmarks, I'd be happy to
contribute measurements to support the KIP.
As a first step, I measured the share group assignor beyond 1,000 members with
the existing ShareGroupAssignorBenchmark, overriding its parameters (trunk
f908227639, SimpleAssignor, full assignment, homogeneous subscription, 10
topics with 10 partitions each; Apple M2 Max, JDK 17, fixed 8 GiB heap with
AlwaysPreTouch, 5 forks):
||Members||Assignment (ms/op)||Time per member (ns)||Allocation per member
(bytes)||
|10,000|1.47|147|907|
|30,000|5.42|181|920|
|60,000|12.59|210|920|
|100,000|25.54|255|931|
Allocated bytes per member stay roughly constant, while the time per member
increases. I haven't isolated the cause of that increase. Measurement details
and tentative observations are in the attached notes; raw results and synthetic
benchmark sources are attached separately. This covers only the assignor, so it
doesn't establish an acceptable limit or the coordinator and broker capacity.
Would coordinator processing and assignment-record volume, or broker
share-session capacity, be the more useful next focus for the KIP?
> Increase the number of consumers in the share consumer group
> ------------------------------------------------------------
>
> Key: KAFKA-21241
> URL: https://issues.apache.org/jira/browse/KAFKA-21241
> Project: Kafka
> Issue Type: Improvement
> Reporter: Prakhar Kulshrestha
> Priority: Major
> Attachments: MapBench.java, WorkingSetBench.java,
> assignor-gc-G1.json, assignor-gc-Parallel.json, assignor-sweep-pretouch.json,
> hashmap-isolated.json, notes.txt, working-set.json
>
>
> Currently the max number of consumers in a share consumer group is default
> to 200 and max to 1000. In bigger platforms where batch processing along
> with slow consumer use cases limiting to just 1000 consumer is very
> restrictive.{*}{*}
> One of the scenario is a batch processing of millions of records and having
> to parallel process 20-30k records in parallel using spring kafka concurrency
> feature would mean we would need at least 20-30k consumers in a consumer
> groups with around 1k pods. Current 1000 limit just let it have ~60pods.
> Request this to be reevaluated and increase the number of consumers that a
> share group can have.
> *group.share.max.size*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
