You are doing something wrong if you need 10k threads to produce 800k messages per second. It feels you are a factor of 1000 off. What size are your messages?
On Thu, Sep 13, 2018, 21:04 Praveen <praveev...@gmail.com> wrote: > Hi there, > > I have a kafka application that uses kafka consumer low-level api to help > us process data from a single partition concurrently. Our use case is to > send out 800k messages per sec. We are able to do that with 4 boxes using > 10k threads and each request taking 50ms in a thread. (1000/50*10000*4) > > I understand that kafka in general uses partitions as its parallelism > model. It is my understanding that if I want the exact same behavior with > kafka streams, I'd need to create 40k partitions for this topic. Is that > right? > > What is the overhead on creating thousands of partitions? If we end up > wanting to send out millions of messages per second, is increasing the > partitions the only way? > > Best, > Praveen >
