Re: latency performance test
Hi Ewen, Thank you for your patient explaining. It is very helpful. Can we assume that the long latency of ProducerPerformance comes from queuing delay in the buffer and it is related to buffer size? Thank you! best, Yuheng On Thu, Jul 16, 2015 at 12:21 AM, Ewen Cheslack-Postava e...@confluent.io wrote: The tests are meant to evaluate different things and the way they send messages is the source of the difference. EndToEndLatency works with a single message at a time. It produces the message then waits for the consumer to receive it. This approach guarantees there is no delay due to queuing. The goal with this test is to evaluate the *minimum* latency. ProducerPerformance focuses on achieving maximum throughput. This means it will enqueue lots of records so it will always have more data to send (and can use batching to increase the throughput). Unlike EndToEndLatency, this means records may just sit in a queue on the producer for awhile because the maximum number of in flight requests has been reached and it needs to wait for responses for those requests. Since EndToEndLatency only ever has one record outstanding, it will never encounter this case. Batching itself doesn't increase the latency because it only occurs when the producer is either a) already unable to send messages anyway or b) linger.ms is greater than 0, but the tests use the default setting that doesn't linger at all. In your example for ProducerPerformance, you have 100 byte records and will buffer up to 64MB. Given the batch size of 8K and default producer settings of 5 in flight requests, you can roughly think of one round trip time handling 5 * 8K = 40K bytes of data. If your roundtrip is 1ms, then if your buffer is full at 64MB it will take you 64 MB / (40 KB/ms) = 1638ms = 1.6s. That means that the record that was added at the end of the buffer had to just sit in the buffer for 1.6s before it was sent off to the broker. And if your buffer is consistently full (which it should be for ProducerPerformance since it's sending as fast as it can), that means *every* record waits that long. Of course, these numbers are estimates, depend on my having used 1ms, but hopefully should make it clear why you can see relatively large latencies. -Ewen On Wed, Jul 15, 2015 at 1:38 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Hi, I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 5000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 buffer.memory=67108864 batch.size=8196 The results show that max latency is 3617ms, avg latency 626.7ms. I wanna know why the latency in producerperformance test is significantly larger than end to end test? Is it because of batching? Are the definitons of these two latencies different? I looked at the source code and I believe the latency is measure for the producer.send() function to complete. So does this latency includes transmission delay, transferring delay, and what other components? Thanks. best, Yuheng -- Thanks, Ewen
Re: latency performance test
That's a good bet. Running with a few different buffer sizes would verify this. Adjusting other settings (e.g. max in flight requests, as mentioned earlier) could also affect latency and throughput. -Ewen On Thu, Jul 16, 2015 at 5:20 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Hi Ewen, Thank you for your patient explaining. It is very helpful. Can we assume that the long latency of ProducerPerformance comes from queuing delay in the buffer and it is related to buffer size? Thank you! best, Yuheng On Thu, Jul 16, 2015 at 12:21 AM, Ewen Cheslack-Postava e...@confluent.io wrote: The tests are meant to evaluate different things and the way they send messages is the source of the difference. EndToEndLatency works with a single message at a time. It produces the message then waits for the consumer to receive it. This approach guarantees there is no delay due to queuing. The goal with this test is to evaluate the *minimum* latency. ProducerPerformance focuses on achieving maximum throughput. This means it will enqueue lots of records so it will always have more data to send (and can use batching to increase the throughput). Unlike EndToEndLatency, this means records may just sit in a queue on the producer for awhile because the maximum number of in flight requests has been reached and it needs to wait for responses for those requests. Since EndToEndLatency only ever has one record outstanding, it will never encounter this case. Batching itself doesn't increase the latency because it only occurs when the producer is either a) already unable to send messages anyway or b) linger.ms is greater than 0, but the tests use the default setting that doesn't linger at all. In your example for ProducerPerformance, you have 100 byte records and will buffer up to 64MB. Given the batch size of 8K and default producer settings of 5 in flight requests, you can roughly think of one round trip time handling 5 * 8K = 40K bytes of data. If your roundtrip is 1ms, then if your buffer is full at 64MB it will take you 64 MB / (40 KB/ms) = 1638ms = 1.6s. That means that the record that was added at the end of the buffer had to just sit in the buffer for 1.6s before it was sent off to the broker. And if your buffer is consistently full (which it should be for ProducerPerformance since it's sending as fast as it can), that means *every* record waits that long. Of course, these numbers are estimates, depend on my having used 1ms, but hopefully should make it clear why you can see relatively large latencies. -Ewen On Wed, Jul 15, 2015 at 1:38 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Hi, I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 5000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 buffer.memory=67108864 batch.size=8196 The results show that max latency is 3617ms, avg latency 626.7ms. I wanna know why the latency in producerperformance test is significantly larger than end to end test? Is it because of batching? Are the definitons of these two latencies different? I looked at the source code and I believe the latency is measure for the producer.send() function to complete. So does this latency includes transmission delay, transferring delay, and what other components? Thanks. best, Yuheng -- Thanks, Ewen -- Thanks, Ewen
Re: latency performance test
The tests are meant to evaluate different things and the way they send messages is the source of the difference. EndToEndLatency works with a single message at a time. It produces the message then waits for the consumer to receive it. This approach guarantees there is no delay due to queuing. The goal with this test is to evaluate the *minimum* latency. ProducerPerformance focuses on achieving maximum throughput. This means it will enqueue lots of records so it will always have more data to send (and can use batching to increase the throughput). Unlike EndToEndLatency, this means records may just sit in a queue on the producer for awhile because the maximum number of in flight requests has been reached and it needs to wait for responses for those requests. Since EndToEndLatency only ever has one record outstanding, it will never encounter this case. Batching itself doesn't increase the latency because it only occurs when the producer is either a) already unable to send messages anyway or b) linger.ms is greater than 0, but the tests use the default setting that doesn't linger at all. In your example for ProducerPerformance, you have 100 byte records and will buffer up to 64MB. Given the batch size of 8K and default producer settings of 5 in flight requests, you can roughly think of one round trip time handling 5 * 8K = 40K bytes of data. If your roundtrip is 1ms, then if your buffer is full at 64MB it will take you 64 MB / (40 KB/ms) = 1638ms = 1.6s. That means that the record that was added at the end of the buffer had to just sit in the buffer for 1.6s before it was sent off to the broker. And if your buffer is consistently full (which it should be for ProducerPerformance since it's sending as fast as it can), that means *every* record waits that long. Of course, these numbers are estimates, depend on my having used 1ms, but hopefully should make it clear why you can see relatively large latencies. -Ewen On Wed, Jul 15, 2015 at 1:38 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Hi, I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 5000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 buffer.memory=67108864 batch.size=8196 The results show that max latency is 3617ms, avg latency 626.7ms. I wanna know why the latency in producerperformance test is significantly larger than end to end test? Is it because of batching? Are the definitons of these two latencies different? I looked at the source code and I believe the latency is measure for the producer.send() function to complete. So does this latency includes transmission delay, transferring delay, and what other components? Thanks. best, Yuheng -- Thanks, Ewen
latency performance test
Hi, I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 5000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 buffer.memory=67108864 batch.size=8196 The results show that max latency is 3617ms, avg latency 626.7ms. I wanna know why the latency in producerperformance test is significantly larger than end to end test? Is it because of batching? Are the definitons of these two latencies different? I looked at the source code and I believe the latency is measure for the producer.send() function to complete. So does this latency includes transmission delay, transferring delay, and what other components? Thanks. best, Yuheng
