Hi Team, We have deployed 150 node Kafka cluster on production for our use case. Recently I have seen issue(s) in Kafka Producer Client.
Use Case: <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies) (Transform) --> Kafka Producer Topology ( Produce it to Multiple Topics) Initially, the data is written to a common buffer topic. The stream topologies will consume from buffer topic and transform from one form to JSON. Finally transformed output goes to Kafka Producer Topology. The nature of the Kafka Producer Topology is to get input from Transform Topology and Write it to stream specific topics. No of Transformation Topologies: 200 No Of Kafka Producer Topologies: 100 Each Kafka Producer Topology will be producing on multiple topics( approx: 2000). Initially, it used to produce to 1000 topics, and now the number of topics has increased hence the problem observed. What is the recommended approach to produce data on multiple topics from multiple clients? Looking forward to your inputs to optimize the Kafka producer topology. Thanks in advance! Client Version: <kafka-client.version>2.3.0</kafka-client.version> Kafka Producer Configuration: retries=3 linger.ms=5 buffer.memory=67108864 batch.size=32768 compression.type=snappy --Senthil
