Very slow producer

2014-12-11 Thread Huy Le Van
Hi, I’m writing my own producer to read from text files, and send line by line to Kafka cluster. I notice that the producer is extremely slow. It's currently sending at ~57KB/node/s. This is like 50-100 times slower than using bin/kafka-console-producer.sh Here’s my producer: final File dir

Re: Very slow producer

2014-12-11 Thread Ewen Cheslack-Postava
Did you set producer.type to async when creating your producer? The console producer uses async by default, but the default producer config is sync. -Ewen On Thu, Dec 11, 2014 at 6:08 AM, Huy Le Van huy.le...@insight-centre.org wrote: Hi, I’m writing my own producer to read from text files,

Re: Very slow producer

2014-12-11 Thread Huy Le Van
Hi Ewen, Thank you for your response. It’s much faster after changing to async. Cheers,Huy, Le Van On Thursday, Dec 11, 2014 at 7:08 p.m., Ewen Cheslack-Postava e...@confluent.io, wrote: Did you set producer.type to async when creating your producer? The console producer uses async