That's a good bet. Running with a few different buffer sizes would verify this. Adjusting other settings (e.g. max in flight requests, as mentioned earlier) could also affect latency and throughput.
-Ewen On Thu, Jul 16, 2015 at 5:20 AM, Yuheng Du <yuheng.du.h...@gmail.com> wrote: > Hi Ewen, > > Thank you for your patient explaining. It is very helpful. > > Can we assume that the long latency of ProducerPerformance comes from > queuing delay in the buffer and it is related to buffer size? > > Thank you! > > best, > Yuheng > > On Thu, Jul 16, 2015 at 12:21 AM, Ewen Cheslack-Postava <e...@confluent.io > > > wrote: > > > The tests are meant to evaluate different things and the way they send > > messages is the source of the difference. > > > > EndToEndLatency works with a single message at a time. It produces the > > message then waits for the consumer to receive it. This approach > guarantees > > there is no delay due to queuing. The goal with this test is to evaluate > > the *minimum* latency. > > > > ProducerPerformance focuses on achieving maximum throughput. This means > it > > will enqueue lots of records so it will always have more data to send > (and > > can use batching to increase the throughput). Unlike EndToEndLatency, > this > > means records may just sit in a queue on the producer for awhile because > > the maximum number of in flight requests has been reached and it needs to > > wait for responses for those requests. Since EndToEndLatency only ever > has > > one record outstanding, it will never encounter this case. > > > > Batching itself doesn't increase the latency because it only occurs when > > the producer is either a) already unable to send messages anyway or b) > > linger.ms is greater than 0, but the tests use the default setting that > > doesn't linger at all. > > > > In your example for ProducerPerformance, you have 100 byte records and > will > > buffer up to 64MB. Given the batch size of 8K and default producer > settings > > of 5 in flight requests, you can roughly think of one round trip time > > handling 5 * 8K = 40K bytes of data. If your roundtrip is 1ms, then if > your > > buffer is full at 64MB it will take you 64 MB / (40 KB/ms) = 1638ms = > 1.6s. > > That means that the record that was added at the end of the buffer had to > > just sit in the buffer for 1.6s before it was sent off to the broker. And > > if your buffer is consistently full (which it should be for > > ProducerPerformance since it's sending as fast as it can), that means > > *every* record waits that long. > > > > Of course, these numbers are estimates, depend on my having used 1ms, but > > hopefully should make it clear why you can see relatively large > latencies. > > > > -Ewen > > > > > > On Wed, Jul 15, 2015 at 1:38 AM, Yuheng Du <yuheng.du.h...@gmail.com> > > wrote: > > > > > Hi, > > > > > > I have run the end to end latency test and the producerPerformance test > > on > > > my kafka cluster according to > > > https://gist.github.com/jkreps/c7ddb4041ef62a900e6c > > > > > > In end to end latency test, the latency was around 2ms. In > > > producerperformance test, if use batch size 8196 to send 50,000,000 > > > records: > > > > > > >bin/kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance > > > speedx1 50000000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 > > > buffer.memory=67108864 batch.size=8196 > > > > > > > > > The results show that max latency is 3617ms, avg latency 626.7ms. I > wanna > > > know why the latency in producerperformance test is significantly > larger > > > than end to end test? Is it because of batching? Are the definitons of > > > these two latencies different? I looked at the source code and I > believe > > > the latency is measure for the producer.send() function to complete. So > > > does this latency includes transmission delay, transferring delay, and > > what > > > other components? > > > > > > > > > Thanks. > > > > > > > > > best, > > > > > > Yuheng > > > > > > > > > > > -- > > Thanks, > > Ewen > > > -- Thanks, Ewen
