Use case scenario: We want to have a fairly low latency, say below 20 ms, and we want to be able to run a few hundred processes (on one machine) both producing and consuming a handful of topics. The throughput is not high, lets say on average 10 messages per second for each process. Most messages are 50-500 bytes large, some may be a few kbytes.
How should we adjust the configuration parameters for our use case? Our experiments so far gives us a good latency but at the expence of CPU utilization. Even with a bad latency, the CPU utilization is not satisfying. Since we will have a lot of processes we are concerned that short poll loops will cause an overconsumption of CPU capacity. We are hoping we might have missed some configuration parameter or that we have some issues with our environment that we can find and solve. We are using both the java client and librdkafka and see similar CPU issues in both clients. We have looked at recommendations from: https://github.com/edenhill/librdkafka/wiki/How-to-decrease-message-latency The only thing that seems to really make a difference for librdkafka is socket.blocking.max.ms, but reducing that also makes the CPU go up. I would really appreciate input on configuration parameters and of any experience with environment issues that has caused CPU load. Or is our scenario not feasible at all? Cheers
