Re: latency test

So, I did my own latency test on a cluster of 3 nodes, and there is a significant difference around the 99%’ile and higher for partitions when measuring the the ack time when configured for a single ack. The graph that I wish I could attach or post clearly shows that around 1/3 of the partitions

Re: latency test

the reason for any other cluster. Its worth noting that this was mostly a latency test over a stress test. There was a single kafka producer object, very small message sizes (100 bytes), and it was only pushing through around 5MB/s worth of data. And the client was configured to minimize the amount

Re: latency test

k, resource > contention on that broker, etc. But it effected that one broker more than > others. And the reasons for my cluster displaying this behavior could be > very different than the reason for any other cluster. > > Its worth noting that this was mostly a latency test over a stress te

Re: latency test

So are you suggesting that the long delays happened in %1 percentile happens in the slower partitions that are further away? Thanks. On Wed, Sep 9, 2015 at 3:15 PM, Helleren, Erik <erik.helle...@cmegroup.com> wrote: > So, I did my own latency test on a cluster o

Re: latency test

u.h...@gmail.com> wrote: > I am running a producer latency test. When using 92 producers in 92 > physical node publishing to 4 brokers, the latency is slightly lower than > using 8 brokers, I am using 8 partitions for the topic. > > I have rerun the test and it gives me the same

Re: latency test

at 10:45 PM, Yuheng Du <yuheng.du.h...@gmail.com> >wrote: > >> I am running a producer latency test. When using 92 producers in 92 >> physical node publishing to 4 brokers, the latency is slightly lower >>than >> using 8 brokers, I am using 8 partitions for

Re: latency test

n > >the > >number of partitions is at least the size of the brokers? > > > >Thanks. > > > >On Thu, Sep 3, 2015 at 10:45 PM, Yuheng Du <yuheng.du.h...@gmail.com> > >wrote: > > > >> I am running a producer latency test. When using 92 producers in

Re: latency test

hroughput first and low latency second. > >>And > >> it does a really good job at both. > >> > >> Disclaimer: I might not like linear algebra, but I do like statistics. > >> Let me know if there are topics that need more explanation above that > >> aren

Re: latency test

so 32 should be > enough. > > As for latency tests, there isn’t a lot of code to do a latency test. If > you just want to measure ack time its around 100 lines. I will try to > push out some good latency testing code to github, but my company is > scared of open sourcing code… s

Re: latency test

I thing the suggestion is to have partitions/brokers >=1, so 32 should be enough. As for latency tests, there isn’t a lot of code to do a latency test. If you just want to measure ack time its around 100 lines. I will try to push out some good latency testing code to github, but my comp

Re: latency test

rik >> >> On 9/4/15, 9:03 AM, "Yuheng Du" <yuheng.du.h...@gmail.com> wrote: >> >> >When I using 32 partitions, the 4 brokers latency becomes larger than >>the >> >8 >> >brokers latency. >> > >> >So is it always true

latency test

I am running a producer latency test. When using 92 producers in 92 physical node publishing to 4 brokers, the latency is slightly lower than using 8 brokers, I am using 8 partitions for the topic. I have rerun the test and it gives me the same result, the 4 brokers scenario still has lower

Re: Latency test

Feng fengta...@gmail.com wrote: I think ProducerPerformance microbenchmark only measure between client to brokers(producer to brokers) and provide latency information. On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Currently, the latency test from kafka

Re: Latency test

Du yuheng.du.h...@gmail.com wrote: Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test the producer to broker and broker to consumer delay seperately? Thanks.

Re: Latency test

I got java out of heap error when running end to end latency test: yuhengdu@consumer0:/packages/kafka_2.10-0.8.2.1$ bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092 192.168.1.1:2181 speedx3 5000 100 1 Exception in thread main java.lang.OutOfMemoryError: Java heap space

Re: Latency test

yuheng.du.h...@gmail.com wrote: Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test the producer to broker and broker to consumer delay seperately? Thanks.

Re: Latency test

I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: bin

Re: Latency test

: Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test the producer to broker and broker to consumer delay seperately? Thanks.

Re: Latency test

wrote: Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test the producer to broker and broker to consumer delay seperately? Thanks.

Re: Latency test

(producer to brokers) and provide latency information. On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test

Re: Latency test

I think ProducerPerformance microbenchmark only measure between client to brokers(producer to brokers) and provide latency information. On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du yuheng.du.h...@gmail.com wrote: Currently, the latency test from kafka test the end to end latency between

Latency test

Currently, the latency test from kafka test the end to end latency between producers and consumers. Is there a way to test the producer to broker and broker to consumer delay seperately? Thanks.