Hi there, We are doing the load test in Kafka with 25tps and first 9 hours it went fine almost 80K/hr messages were processed after that we see a lot of lags and we stopped the incoming load.
Currently we see 15K/hr messages are processing. We have 40 consumer instances with concurrency 4 and 2 topics and both is having 160 partitions so each consumer with each partition. What we found that some of the partitions are sitting idle and some of are overloaded and its really slowing down the consumer message processing. Why rebalancing is not happening and existing messages are not distributed equally among the instances? We tried to restart the app still the same pace. Any idea what could be the reason? Thanks Achintya
