Hi, I am new to kafka. I am getting less throughput and high latency in publishing message of size 100-200 bytes.
I have the producer configured with the following configuration. I am using akka-reactive kafka to publish messages Configuration: kafka { producer { parallelism = 500 bootstrap.servers = "x:9092" close-timeout = 86400s auto.offset.reset = "earliest" max.redeliveries = 0 use-dispatcher = "kafka-dispatcher" kafka-clients { retries = 0 acks = 0 buffer.memory=67108864 batch.size=8192 linger.ms = 2 } } } kafka-dispatcher { type = Dispatcher executor = "fork-join-executor" fork-join-executor { parallelism-min = 10 parallelism-factor = 20.0 parallelism-max = 300 } throughput = 1 } Traffic Pattern: Message published to 10 Topic with 150 partitions. I am sending the request for 10 Topics with 30 concurrent requests from Jmeter. I am getting the throughput of 3-5 Hits/sec. Kafka Server Configuration: Kafka server and Zookeeper on the same machine centos with 4 CPU 100GB Diskspace and 16 GB RAM. All are defaults except the below one: # The number of threads that the server uses for receiving requests from the network and sending responses to the network num.network.threads=50 # The number of threads that the server uses for processing requests, which may include disk I/O num.io.threads=100 Let me know how to increase the throughput of producer ? Regards, Gnana