I believe you have answered your own question. Having the system enforce which topics each consumer in a consumer group needs to consume messages from may be restricting, and possibly expensive. I have seen this brought up a few times and discussed it with a colleague. I am not clear as to what the advantage is to have the members of a consumer group consume from different topics. I have seen instances where this was caused by a misconfiguration, but are there cases where this would be done intentionally? If it doesn't exist already, maybe mismatched topics in a consumer group should be alerted on?
On Tue, Apr 3, 2018 at 11:18 AM Gabriel Giussi <gabrielgiu...@gmail.com> wrote: > Kafka brokers version: 0.11.0.0 > Kafka client version: 0.11.0.2 > > If we have two KafkaConsumer using the same group.id (running in the same > process or in two different processes) and one of them is closed, it > triggers a rebalance in the other KafkaConsumer even if they were > subscribed to different topics. > I suppose that brokers must be taking into account only the group.id for a > rebalance and not the subscribed topics corresponding to the pair > (group_id,member_id) of the LeaveGroupRequest but I'm wondering if this is > the expected behavior or it's something that should be improved? > > I guess that is probably the first option to avoid a more complex rebalance > in the broker and considering that the solution is very simple, i.e. just > use different group ids for different KafkaConsumer that subscribe to > different topics even if they are running in the same process. > > Thanks. >
