[
https://issues.apache.org/jira/browse/KAFKA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744874#comment-16744874
]
ASF GitHub Bot commented on KAFKA-7641:
---------------------------------------
stanislavkozlovski commented on pull request #6163: KAFKA-7641: Introduce
"consumer.group.max.size" config to limit consumer group sizes
URL: https://github.com/apache/kafka/pull/6163
This patch introduces a new config - "consumer.group.max.size", which caps
the maximum size any consumer group can reach. It has a default value of
Int.MAX_VALUE.
Once a consumer group is of the maximum size, subsequent JoinGroup requests
receive a MAX_SIZE_REACHED error.
In the case where the config is changed and a Coordinator broker with the
new config loads an old group that is over the threshold, members are kicked
out of the group and a rebalance is forced.
I have added two integration tests for both scenarios - a member joining an
already-full group and a rolling restart with a new config
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add `consumer.group.max.size` to cap consumer metadata size on broker
> ---------------------------------------------------------------------
>
> Key: KAFKA-7641
> URL: https://issues.apache.org/jira/browse/KAFKA-7641
> Project: Kafka
> Issue Type: Improvement
> Reporter: Boyang Chen
> Assignee: Stanislav Kozlovski
> Priority: Major
>
> In the JIRA discussion https://issues.apache.org/jira/browse/KAFKA-7610,
> Jason concluded an edge case of current consumer protocol which could cause
> memory burst on broker side:
> ```the case we observed in practice was caused by a consumer that was slow to
> rejoin the group after a rebalance had begun. At the same time, there were
> new members that were trying to join the group for the first time. The
> request timeout was significantly lower than the rebalance timeout, so the
> JoinGroup of the new members kept timing out. The timeout caused a retry and
> the group size eventually become quite large because we could not detect the
> fact that the new members were no longer there.```
> Since many disorganized join group requests are spamming the group metadata,
> we should define a cap on broker side to avoid one consumer group from
> growing too large. So far I feel it's appropriate to introduce this as a
> server config since most times this value is only dealing with error
> scenarios, client users shouldn't worry about this config.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
