Re: Reduce latency

you can configure that, to block or to fail: http://kafka.apache.org/documentation.html#producerconfigs By default it should block On Tue, Aug 18, 2015 at 4:57 PM, Yuheng Du wrote: > I see. So the internal queue overwrites the producer buffer size > configuration? When buffer is full the produc

Re: Reduce latency

I see. So the internal queue overwrites the producer buffer size configuration? When buffer is full the producer will block sending, right? On Tue, Aug 18, 2015 at 3:52 PM, Tao Feng wrote: > From what I understand, if you set the throughput to -1, the > producerperformance will push records as m

Re: Reduce latency

>From what I understand, if you set the throughput to -1, the producerperformance will push records as much as possible to an internal per topic per partition queue. In the background there is a sender IO thread handling the actual record sending process. If you push record to the queue faster than

Re: Reduce latency

I see. Thank you Tao. But now I don't get it what Jay said that my latency test only makes sense if I set a fixed throughput. Why do I need to set a fixed throughput for my test instead of just set the expected throughput to be -1 (as much as possible)? Thanks. On Tue, Aug 18, 2015 at 2:43 PM, Ta

Re: Reduce latency

Hi Yuheng, The 1 record/s is just a param for producerperformance for your producer target tput. It only takes effect to do the throttling if you tries to send more than 1 record/s. The actual tput of the test depends on your producer config and your setup. -Tao On Tue, Aug 18, 2015 at

Re: Reduce latency

Also, When I set the target throughput to be 1 records/s, The actual test results show I got an average of 579.86 records per second among all my producers. How did that happen? Why this number is not 1 then? Thanks. On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du wrote: > Thank you Jay, tha

Re: Reduce latency

Thank you Jay, that really helps! Kishore, Where you can monitor whether the network is busy on IO in visual vm? Thanks. I am running 90 producer process on 90 physical machines in the experiment. On Tue, Aug 18, 2015 at 1:19 AM, Jay Kreps wrote: > Yuheng, > > From the command you gave it looks

Re: Reduce latency

Yuheng, >From the command you gave it looks like you are configuring the perf test to send data as fast as possible (the -1 for target throughput). This means it will always queue up a bunch of unsent data until the buffer is exhausted and then block. The larger the buffer, the bigger the queue. T

Re: Reduce latency

Just to clarify it is 1 thread / Broker / Producer. Javadoc recommends to use one Producer across multiple threads and batching is done behind the scenes so throughput should not be a problem. So if you have more brokers, the same Producer instance will use multiple threads to send to each Broker,

Re: Reduce latency

If you run producerPerformance test, there is only one thread per KafkaProducer doing the actual sending. But one network request could contain multiple batches from what I understand. On Mon, Aug 17, 2015 at 5:42 PM, Yuheng Du wrote: > Thank you Kishore, I made the buffer twice the size of the

Re: Reduce latency

Thank you Kishore, I made the buffer twice the size of the batch size and the latency has reduced significantly. But is there only one thread io thread sending the batches? Can I increase the number of threads sending the batches so more than one batch could be sent at the same time? Thanks. O

Re: Reduce latency

Your batch.size is 8196 and your buffer.memory is 67108864. This means 67108864/8196 ~ 8188 batches are in memory ready to the sent. There is only one thread io thread sending them. I would guess that the io thread ( kafka-producer-network-thread) would be busy. Please check it in visual vm. In st

Re: Reduce latency

Yes there is. But if we are using ProducerPerformance test, it's configured as giving input when running the test command. Do you write a java program to test the latency? Thanks. On Thu, Aug 13, 2015 at 3:54 PM, Alvaro Gareppe wrote: > I'm using last one, but not using the ProducerPerformance,

Re: Reduce latency

I'm using last one, but not using the ProducerPerformance, I created my own. but I think there is a producer.properties file in config folder in kafka.. is that configuration not for this tester ? On Thu, Aug 13, 2015 at 4:18 PM, Yuheng Du wrote: > Thank you Alvaro, > > How to use sync producers

Re: Reduce latency

Thank you Alvaro, How to use sync producers? I am running the standard ProducerPerformance test from kafka to measure the latency of each message to send from producer to broker only. The command is like "bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 5000 100

Re: Reduce latency

Are you measuring latency as time between producer and consumer ? In that case, the ack shouldn't affect the latency, cause even tough your producer is not going to wait for the ack, the consumer will only get the message after its commited in the server. About latency my best result occur with s

Reduce latency

I am running an experiment where 92 producers is publishing data into 6 brokers and 10 consumer are reading online data simultaneously. How should I do to reduce the latency? Currently when I run the producer performance test the average latency is around 10s. Should I disable log.flush? How to d

Re: Reduce latency

Also, the latency results show no major difference when using ack=0 or ack=1. Why is that? On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du wrote: > I am running an experiment where 92 producers is publishing data into 6 > brokers and 10 consumer are reading online data simultaneously. > > How should