Hi again, I finally found out what to change to get the desired topic/partition balancing. I had to set the Consumer property
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor As I found out this is the recommended setting when using MirrorMaker and in fact I have a MirrorMaker Job with this configuration which worked as expected. So does my application now ;-) Best regards, Franz Gesendet: Mittwoch, 27. Februar 2019 um 10:54 Uhr Von: "Franz van Betteraey" <[email protected]> An: [email protected] Betreff: Load balance Kafka record consumption when using topic pattern Hi all, I habe a question: If two KafkaConsumer which uses the same consumer group (group.id) subscribe to the same topicPattern - will Kafka than load balance the resolved topics (! - not only partitions) between the two processes? In my case about 400 topics match the pattern and each topic has one partion. I wonder if I can load balance the consumption of the topics via different processes/threads with consumers that belong to the same consumer group but it seems that only one consumer handles all the topics (and thus all the partitions). Is this because only partitions are load balanced - or maybe because a topic pattern is used by the subscription? I made some deeper debugging. My test setup is two threads with an own Kafka Consumer. The Kafka Consumer share the same goupId. Debug printing the topic description outputs something like this (IPs are obfuscated): (groupId=foo-bar-group, isSimpleConsumerGroup=false, members= (memberId=KafkaTestApplication:KafkaTestApplication-1-20126d55-678d-46f5-bc73-3769db2c8901, clientId=KafkaTestApplication:KafkaTestApplication-1, host=/aa.bb.c.dd, assignment=(topicPartitions= <all topics here>)), (memberId=KafkaTestApplication:KafkaTestApplication-2-8d34e81a-8f57-4d4e-bd9e-b56edc4e706c, clientId=KafkaTestApplication:KafkaTestApplication-2, host=/aa.bb.c.dd, assignment=(topicPartitions=)), partitionAssignor=range, state=Stable, coordinator=aaa.bb.ccc.d:9092 (id: 9 rack: null) ) I can see that I have two different members in the consumer group with two different clientIds. Nevertheless all topics are assigned to the first consumer and the second consumer has no topics assigned. I expected that both members get some topics. Both members poll every few seconds. I also ask this question on StaockOverflow: https://stackoverflow.com/questions/54594544/load-balance-kafka-record-consumption-when-using-topic-pattern Any information to understand the behaviour would be of great help for me. Thanks in advance and best regards Franz
