Hi Arvind Yes, there is data to support the claim that sharing a single producer instance across threads in Apache Kafka is generally faster than having multiple instances.
The reason for this is that a single producer instance can take advantage of batch processing, which allows it to send multiple messages to Kafka in a single network request. This reduces the network overhead and can result in higher throughput. On the other hand, if multiple producer instances are used, each instance will have its own network connection to Kafka, resulting in higher network overhead. Additionally, each instance will have to manage its own queue of messages to send, which can increase the overall processing overhead. Several benchmark tests have been conducted to compare the performance of single versus multiple producer instances. For example, one test conducted by Confluent found that using a single producer instance was up to 50% faster than using multiple instances. Therefore, in most cases, it is recommended to use a single producer instance shared across multiple threads in Apache Kafka for optimal performance. Thank you Santhosh Gopal On Mon, Apr 24, 2023 at 9:01 AM Arvid Sundbom <arvid.sundbo...@gmail.com> wrote: > Hi! > I have a question about executing a Kafka producer application, utilizing > multiple threads. > In the documentation for Kafka producers ( > > https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html > ) > it says that "...sharing a single producer instance across threads will > generally be faster than having multiple instances.". > > I am mainly wondering if there is any data to verify this claim? > The reason for this is that I have carried out rather extensive performance > tests, with varying levels of computational load per message produced, > using between 1 and 24 threads, and it seems that there is really no > situation in which a single, shared producer achieves a higher performance > than > if each executing thread is assigned its own producer instance. > > Kind regards, > Arvid >
