[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325234#comment-17325234
]
Guozhang Wang commented on KAFKA-12675:
---------------------------------------
Wow this is super! Thanks [~showuon] [~twmb], please ping me as well as
[~ableegoldman] when you think https://github.com/apache/kafka/pull/10552 is
ready to review.
> Improve sticky general assignor scalability and performance
> -----------------------------------------------------------
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
> Reporter: Luke Chen
> Assignee: Luke Chen
> Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#0000ff}500{color};
> partitionCount = {color:#0000ff}2000{color};
> consumerCount = {color:#0000ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#0000ff}50{color};
> partitionCount = 1{color:#0000ff}000{color};
> consumerCount = 1{color:#0000ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#0000ff}50{color};
> partitionCount = 8{color:#0000ff}00{color};
> consumerCount = 8{color:#0000ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
