Jay log.flush.interval=10000 log.default.flush.interval.ms=1000
I used boxes having - 8 core Intel(R) Xeon(R) CPU E5405 @ 2.00GHz - 16G RAM Jun, Created a jira issue, https://issues.apache.org/jira/browse/KAFKA-399 There was no compression at the producer. Thanks Min 2012/7/11 Jun Rao <jun...@gmail.com>: > Also, did you enable compression in the producer? > > Thanks, > > Jun > > On Mon, Jul 9, 2012 at 11:53 PM, Min <mini...@gmail.com> wrote: > >> Hello, >> >> I tested again and I could reproduce the situation. >> >> I run 0.7.0 with the following shell command >> ./kafka-0.7.0-incubating-src/bin/kafka-server-start.sh >> config/server.properties >> then 0.7.1 >> ./kafka-0.7.1-incubating/bin/kafka-server-start.sh config/server.properties >> >> configuration values in the server.properties are all default but the >> ZK address. >> >> My producer and broker were running on different machines in same rack. >> They are not VMs but physical machines. Also I don't feel some other >> process interfere my test. >> >> >> I'm attaching the dstat result and my producer code. >> Producer is so simple that read files in a given directory by line and >> send every 1000 lines to the broker. >> >> Thanks >> Min >> >> ./kafka-0.7.0-incubating-src/bin/kafka-server-start.sh >> config/server.properties >> (Producer) >> ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- >> usr sys idl wai hiq siq| read writ| recv send| in out | int csw >> 23 2 72 2 0 1| 88M 896k| 98k 50M| 0 0 |2855 2627 >> 20 1 77 1 0 1| 81M 1144k| 79k 48M| 0 0 |2858 2423 >> 21 1 77 0 0 1| 85M 0 | 77k 47M| 0 0 |2655 2222 >> 6 0 93 0 0 0| 24M 240k| 38k 15M| 0 0 |1664 1232 >> 18 1 80 0 1 1| 71M 0 | 72k 42M| 0 0 |2657 2152 >> 21 1 69 7 0 1| 75M 1704k| 92k 48M| 0 0 |2959 2872 >> 20 2 52 25 0 1| 90M 512k| 82k 48M| 0 0 |2884 2940 >> 20 1 78 0 0 0| 85M 0 | 77k 47M| 0 0 |2800 2459 >> 22 2 75 0 1 1| 88M 320k| 81k 51M| 0 0 |2840 2590 >> 24 1 74 0 1 0| 90M 0 | 88k 52M| 0 0 |2819 2526 >> 23 1 71 2 0 1| 87M 1408k| 96k 51M| 0 0 |2924 2661 >> 23 2 73 1 1 1| 89M 744k| 82k 51M| 0 0 |2851 2485 >> 25 1 72 0 1 1| 94M 0 | 89k 55M| 0 0 |2941 2545 >> 22 1 76 0 0 1| 88M 240k| 81k 50M| 0 0 |2804 2581 >> 23 1 75 0 0 1| 93M 0 | 83k 53M| 0 0 |2868 2530 >> 23 1 69 5 0 1| 87M 1400k| 100k 53M| 0 0 |2959 2668 >> >> (Broker 0.7.0) >> usr sys idl wai hiq siq| read writ| recv send| in out | int csw >> 4 3 86 4 0 2|2056k 197M| 51M 68k| 0 0 | 11k 1911 >> 5 3 86 4 0 2|2048k 209M| 53M 72k| 0 0 | 11k 2097 >> 5 3 85 4 0 2|2056k 207M| 52M 77k| 0 0 | 11k 2084 >> 4 4 86 4 0 2|2048k 208M| 53M 71k| 0 0 | 11k 2101 >> 5 3 86 4 0 2|2064k 201M| 51M 68k| 0 0 | 11k 2055 >> 4 4 86 4 0 2|2048k 207M| 53M 71k| 0 0 | 11k 2007 >> 5 4 86 4 0 2|2048k 209M| 53M 71k| 0 0 | 11k 2018 >> 5 3 85 4 0 2|2056k 207M| 52M 76k| 0 0 | 11k 1997 >> 4 4 85 5 0 2|2056k 197M| 50M 67k| 0 0 | 11k 1907 >> 5 4 86 4 0 2|2056k 205M| 52M 71k| 0 0 | 11k 2053 >> 5 3 86 4 0 2|2048k 205M| 52M 70k| 0 0 | 11k 2000 >> >> ./kafka-0.7.1-incubating/bin/kafka-server-start.sh config/server.properties >> >> (Producer) >> ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- >> usr sys idl wai hiq siq| read writ| recv send| in out | int csw >> 9 1 89 1 0 0| 41M 416k| 51k 27M| 0 0 |2205 1618 >> 9 1 88 2 0 0| 49M 0 | 47k 27M| 0 0 |2212 1952 >> 8 1 89 1 0 0| 44M 0 | 58k 26M| 0 0 |2201 1713 >> 10 1 87 2 0 0| 47M 256k| 57k 26M| 0 0 |2199 1992 >> 9 1 84 5 0 1| 46M 2488k| 56k 27M| 0 0 |2472 1924 >> 9 1 89 1 0 0| 42M 384k| 53k 26M| 0 0 |2187 1730 >> 9 1 89 1 0 0| 48M 0 | 44k 27M| 0 0 |2189 1691 >> 10 1 88 1 0 0| 45M 0 | 58k 27M| 0 0 |2205 1884 >> 9 1 89 1 0 0| 48M 224k| 58k 27M| 0 0 |2225 1715 >> 9 1 76 12 0 1| 44M 2496k| 57k 26M| 0 0 |2425 2133 >> 9 1 88 2 0 0| 45M 584k| 51k 26M| 0 0 |2194 1685 >> 9 1 89 1 0 0| 46M 0 | 43k 25M| 0 0 |2166 1727 >> 9 1 90 1 0 0| 47M 0 | 54k 25M| 0 0 |2183 1629 >> >> (Broker - 0.7.1) >> ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- >> usr sys idl wai hiq siq| read writ| recv send| in out | int csw >> 9 2 86 2 0 1|2048k 103M| 26M 36k| 0 0 |6376 1849 >> 8 2 86 2 0 1|2056k 104M| 26M 37k| 0 0 |6431 1945 >> 9 2 86 2 0 1|2048k 101M| 26M 42k| 0 0 |6431 1884 >> 9 2 87 2 0 1|2048k 104M| 26M 35k| 0 0 |6300 1561 >> 9 2 86 2 0 1|2048k 104M| 27M 37k| 0 0 |6542 1543 >> 9 2 86 2 0 1|2056k 104M| 26M 37k| 0 0 |6487 1638 >> 9 2 87 2 0 1|2048k 101M| 26M 36k| 0 0 |6402 1721 >> 9 2 86 2 0 1|2048k 104M| 26M 43k| 0 0 |6436 1762 >> 9 2 86 2 0 1|2048k 106M| 27M 37k| 0 0 |6527 1799 >> 9 2 86 3 0 1|2056k 101M| 26M 36k| 0 0 |6378 1521 >> 9 2 86 2 0 1|2048k 104M| 27M 37k| 0 0 |6521 1568 >> 9 2 87 2 0 1|2048k 104M| 26M 37k| 0 0 |6452 1708 >> >> >> >> >> 2012/7/9 Jay Kreps <jay.kr...@gmail.com>: >> > Not that is known to me. Is this something you can reproduce? Are the >> > settings the same on the two brokers? Can you check iostat between the >> two >> > and see if the change is due to disk activity or cpu? >> > >> > -Jay >> > >> > On Sun, Jul 8, 2012 at 12:03 AM, Min <mini...@gmail.com> wrote: >> > >> >> Hello, >> >> >> >> Recently I've upgrade to 0.7.1. But 0.7.1 didn't show performance as >> >> 0.7.0. >> >> >> >> With 0.7.0 I can push 45MBps to the broker. But I only can push 27MBps >> >> with 0.7.1. >> >> >> >> Is there any a fetch to affect the performance? >> >> >> >> Thanks >> >> >>
