Hi, sorry for that, I sent my original email to this list by mistake (gmail autocomplete fooled me), the page I linked isn't open to buplic.
Anyway, since you are interested, here is the sample commands and output from a VirtualBox image on my laptop. 1. Create a topic kafka-topics.sh --create --zookeeper localhost:2181 --partitions 2 --replication-factor 1 --topic perf_test_2_partitions 2. Write two millions messages, each 1KB, at a target rate of 50MB/s: $ kafka-producer-perf-test.sh --topic perf_test_2_partitions --num-records 2000000 --record-size 1024 --throughput 50000 --producer-props bootstrap.servers=localhost:9092 249934 records sent, 49986.8 records/sec (48.82 MB/sec), 85.7 ms avg latency, 415.0 max latency. 250142 records sent, 50028.4 records/sec (48.86 MB/sec), 2.2 ms avg latency, 33.0 max latency. 250029 records sent, 50005.8 records/sec (48.83 MB/sec), 4.6 ms avg latency, 74.0 max latency. 249997 records sent, 49999.4 records/sec (48.83 MB/sec), 16.2 ms avg latency, 98.0 max latency. 249902 records sent, 49980.4 records/sec (48.81 MB/sec), 5.7 ms avg latency, 95.0 max latency. 249568 records sent, 49893.6 records/sec (48.72 MB/sec), 1.8 ms avg latency, 21.0 max latency. 248190 records sent, 49638.0 records/sec (48.47 MB/sec), 10.9 ms avg latency, 158.0 max latency. 2000000 records sent, 49993.750781 records/sec (48.82 MB/sec), 17.06 ms avg latency, 415.00 ms max latency, 2 ms 50th, 81 ms 95th, 343 ms 99th, 407 ms 99.9th 3. Read ten million messages (each 1KB), at a rate of 231MB/s (it was physical I/O - saw with iostat command that the SSD did work at that rate): (I ran the previous command several times, so the topic did have 10M messages) $ kafka-consumer-perf-test.sh --topic perf_test_2_partitions --messages 10000000 --broker-list localhost:9092 --date-format HH:mm:ss:SSS start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 19:32:53:408, 19:33:35:523, 9765.6250, 231.8800, 10000000, 237445.0908 If you / someone has followup questions, might be best to do it off Spark dev list... Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Thu, Jun 1, 2017 at 7:25 AM, 郭健 <guo.j...@immomo.com> wrote: > It seems an internal page so I cannot access it: > > Your email address doesn't have access to *equalum.atlassian.net > <http://equalum.atlassian.net>* > > > > *发件人**: *Ofir Manor <ofir.ma...@equalum.io> > *日期**: *2017年5月26日 星期五 01:12 > *至**: *dev <dev@spark.apache.org> > *主**题**: *FYI - Kafka's built-in performance test tool > > > > comes with source code. > Some basic results from the VM, > > - Write every second 50K-60K messages, each 1KB (total 50MB-60MB) > - Read every second more than 200K messages, each 1KB. > > May help in assessing whether any Kafka-related slowness is Kafka > limitation or our implementation. > > > > https://equalum.atlassian.net/wiki/display/EQ/Kafka+ > performance+test+tool > > Ofir Manor > > Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io >
