Hello,
We are planning to use Apache Kafka in our system to replace local MSMQs (.net message queues). *Single Environment* Currently, we have about 100 message queues per environment. In Kafka, we plan to use approximately 20 topics to accommodate the 100 message queues. We will have 100 consumer groups (1 per each message queue) that read from these topics. For example, existing queues A, B, C, D will be replaced by a single Kafka Topic. There will be 4 consumer groups that read from this topic. Each consumer group will receive all messages in the topic but will only process the messages for its assigned queue. It will discard the other messages. *One Data Center* In our current deployment architecture, we host multiple environments in a single data center. (About 100 environments in one data center) We plan to use one (or a few) Kafka cluster for one data center. With this architecture, one Kafka cluster will have about 2K topics (20 per environment X 100 environments). And there will be about 10K consumer groups. Can you kindly provide your recommendations and best practices for this use-case and scale particularly around the below 1. Are there any limits on the number of topics? Is it okay to have 2K topics in a Kafka cluster? 2. Are there any limits on the number of consumer groups? Is it okay to have 10K consumer groups in a cluster? 3. The consumer groups will be discarding a lot of messages meant for other message queues. Are there any recommended approach for this scenario or ways to avoid it? Thank You, Akshay Dhokale
