Hey Lahiru, As the log message describes, the member is leaving the group because the consumer has been closed by your process/application. You need to track down where in your application code the Consumer#close method is being called, and why.
There's no technical limit on how many consumers can be in the same process. Of course there is a practical limit, eg if you try to start up millions of consumers then they may struggle to stay in the group for example due to the millions of background heartbeat threads competing for cpu and network resources. But I see no reason why a modern machine would be unable to handle just 4 consumers, and anyways the log message you provided makes it clear that the consumer is only leaving the group due to being manually closed. Hope that helps, Sophie On Thu, May 6, 2021 at 7:29 AM Lahiru Chandima <[email protected]> wrote: > Hi, > > I am trying to place multiple kafka consumers of a single consumer group in > the same process. But, I am unable to put more than 3 consumers in the same > process.The topic which the consumer group tries to read has 10 > partitions.When I try to attach the 4th consumer, it leaves the group after > the first poll, with below message. No partitions get assigned to the > consumer. > > [ForkJoinPool.commonPool-worker-3] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Consumer > clientId=consumer-group65-4, groupId=group65] Member > consumer-group65-4-b148f7ec-b852-46ed-8514-c3af3bd78395 sending LeaveGroup > request to coordinator my-cluster:9092 (id: 2147483645 rack: null) due to > the consumer is being closed > > What is the reason for this? Is there any way to put more than 3 consumers > in the same process? > > Thank you > -- > Lahiru Chandima >
