Hi all, I wanted to understand, having large numbers of consumers on producer latency and brokers. I have around 7K independent consumers. Each consumer is consuming all partitions of a topic. I have manually assigned partitions of a topic to a consumer, not using consumer groups. Each consumer is consuming msg only one topic. Message size is small less than a KB. Produce throughput is also low. But when produce spike come then produce latency (here ack=1). Brokers resource is also very low. I want to understand the impact on having large number of consumers on Kafka.
*Cluster Details: * Broker#: 13 (1 Broker : 14 cores & 36GB memory ) Kafka cluster version: 2.0.0 Kafka Java client version: 2.0.0 Number topics: ~15. Number of consumers: 7K (all independent and manually assigned all partitions of a topic to a consumers. One consumer is consuming all partitions from a topic only) Regards, Hrishikesh
