Hi all,
I habe a question: If two KafkaConsumer which uses the same consumer group
(group.id) subscribe to the same topicPattern - will Kafka than load balance
the resolved topics (! - not only partitions) between the two processes?
In my case about 400 topics match the pattern and each topic has one partion. I
wonder if I can load balance the consumption of the topics via different
processes/threads with consumers that belong to the same consumer group but it
seems that only one consumer handles all the topics (and thus all the
partitions). Is this because only partitions are load balanced - or maybe
because a topic pattern is used by the subscription?
I made some deeper debugging. My test setup is two threads with an own Kafka
Consumer. The Kafka Consumer share the same goupId.
Debug printing the topic description outputs something like this (IPs are
obfuscated):
(groupId=foo-bar-group, isSimpleConsumerGroup=false,
members=
(memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901,
clientId=KafkaTestApplication:KafkaTestApplication-1,
host=/aa.bb.c.dd,
assignment=(topicPartitions= <all topics here>)),
(memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c,
clientId=KafkaTestApplication:KafkaTestApplication-2,
host=/aa.bb.c.dd,
assignment=(topicPartitions=)),
partitionAssignor=range,
state=Stable,
coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null)
)
I can see that I have two different members in the consumer group with two
different clientIds. Nevertheless all topics are assigned to the first consumer
and the second consumer has no topics assigned. I expected that both members
get some topics. Both members poll every few seconds.
I also ask this question on StaockOverflow:
https://stackoverflow.com/questions/54594544/load-balance-kafka-record-consumption-when-using-topic-pattern
Any information to understand the behaviour would be of great help for me.
Thanks in advance and best regards
Franz
