Ah, but what is the cpu utilization? Can you run "top" or something during the test? My suspicion is that the producer is maybe just cpu bound.
-Jay On Wed, Mar 28, 2012 at 3:57 AM, 刘明敏 <diveintotomor...@gmail.com> wrote: > 4 cpus on Procuder: > > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 30 > model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz > stepping : 5 > cpu MHz : 2393.580 > cache size : 8192 KB > physical id : 0 > siblings : 4 > core id : 0 > cpu cores : 4 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 > clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm > constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc > aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm > sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid > bogomips : 4787.16 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 1 > vendor_id : GenuineIntel > cpu family : 6 > model : 30 > model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz > stepping : 5 > cpu MHz : 2393.580 > cache size : 8192 KB > physical id : 0 > siblings : 4 > core id : 1 > cpu cores : 4 > apicid : 2 > initial apicid : 2 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 > clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm > constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc > aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm > sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid > bogomips : 4787.17 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 2 > vendor_id : GenuineIntel > cpu family : 6 > model : 30 > model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz > stepping : 5 > cpu MHz : 2393.580 > cache size : 8192 KB > physical id : 0 > siblings : 4 > core id : 2 > cpu cores : 4 > apicid : 4 > initial apicid : 4 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 > clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm > constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc > aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm > sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid > bogomips : 4787.17 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: > > processor : 3 > vendor_id : GenuineIntel > cpu family : 6 > model : 30 > model name : Intel(R) Xeon(R) CPU X3430 @ 2.40GHz > stepping : 5 > cpu MHz : 2393.580 > cache size : 8192 KB > physical id : 0 > siblings : 4 > core id : 3 > cpu cores : 4 > apicid : 6 > initial apicid : 6 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 > clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm > constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc > aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm > sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid > bogomips : 4787.17 > clflush size : 64 > cache_alignment : 64 > address sizes : 36 bits physical, 48 bits virtual > power management: > > === > not sure if I am sending the right place this time. > > 2012/3/28 Jay Kreps <jay.kr...@gmail.com> > > > What does CPU look like on the producer? > > > > Also, can you please use the apache mailing list (cc'd). This google > group > > is from our pre-apache days and is long dead. > > > > -Jay > > > > On Tue, Mar 27, 2012 at 7:42 PM, Kafka > > <kafka-dev+msga...@googlegroups.com>wrote: > > > > > A message has been sent to the kafka-dev< > > http://groups.google.com/group/kafka-dev?hl=en_US>group and is awaiting > > approval. We've included this message for your review. > > > > > > The message requires moderation because the user does not have > permission > > > to post. > > > > > > You can approve or reject< > > http://groups.google.com/group/kafka-dev/pendmsg?hl=en_US>this message > or > > you can approve this message by replying to this email. > > > [image: Visit Google Groups] > > > > > > Start <https://groups.google.com/groups/create?hl=en_US&src=email> > your > > > own group, visit <http://groups.google.com/support/?hl=en_US&src=email > >the > > help center, or > > > report <https://groups.google.com/groups/abuse?hl=en_US&src=email> > > abuse. > > > > > > > > > ---------- Forwarded message ---------- > > > From: 刘明敏 <diveintotomor...@gmail.com> > > > To: kafka-users@incubator.apache.org > > > Cc: kafka-...@googlegroups.com > > > Date: Wed, 28 Mar 2012 10:42:11 +0800 > > > Subject: Re: [kafka-dev] performance of async producer stops increase > > when > > > broker is more than 2 > > > I have 1 producer on 1 machine. > > > > > > The perf of sync producer is : > > > > > > 1 borker: > > > *20.3277（M/s）*,*213151.4441(#/s)* (sync) > > > 2 borkers: > > > *39.5995*, *415230.6606* (sync) > > > 3 borkers: > > > *58.0836*, *609050.4903*(sync) > > > > > > compared with async producer : > > > when 1 broker: > > > 20.3164 M/s,213033.3823#/s (async) > > > > > > when 2 brokers: > > > 37.4754 M/s,392958.1892#/s (async) > > > > > > when 3 brokers(perf stops increase) > > > 36.6925 M/s, 384748.5668#/s (async) > > > > > > the sync producer reached 600K #/s(far beyond sync producer's 384k #/s) > > So > > > I doubt if it is the case of maxing out network or procuder. > > > > > > === > > > and,I post the thread from google group: > > > https://groups.google.com/forum/?fromgroups#!forum/kafka-dev > > > > > > On Wed, Mar 28, 2012 at 3:25 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > > > > > >> Hi, > > >> > > >> How many producers are you using and on how many machines? Also what > is > > >> the > > >> network speed (1gbit?) Likely you are maxing out either the producer > > >> machine(s) or the network itself. > > >> > > >> Also, quick question, how did you come across this address? This > address > > >> has been dead for a while now, and the new mailing list is on Apache > so > > I > > >> just want to make sure we don't have a dangling pointer somewhere. > Would > > >> be > > >> good to move the discussion to the Apache list so others can find it. > > >> > > >> -Jay > > >> > > >> On Tue, Mar 27, 2012 at 2:10 AM, mmLiu <diveintotomor...@gmail.com> > > >> wrote: > > >> > > >> > I tested the performance of producer with > > >> ./bin/kafka-producer-perf-test.sh > > >> > > > >> > when 1 broker: > > >> > 20.3164 M/s,213033.3823#/s (async) > > >> > > > >> > when 2 brokers: > > >> > 37.4754 M/s,392958.1892#/s (async) > > >> > > > >> > when 3 brokers(perf stops increase) > > >> > 36.6925 M/s, 384748.5668#/s (async) > > >> > > > >> > when 4 brokers(even decrease): > > >> > 33.2244 M/s, 348383.5006 #/s (async) > > >> > > > >> > > > >> > Any idea why? > > >> > > > >> > FYI:the performance of sync producer increases(almost) linearly as > > >> number > > >> > of brokers increases. > > >> > > > > > > > > > > > > -- > > > Best Regards > > > > > > ---------------------- > > > 刘明敏 | mmLiu > > > > > > > > > > > > > > > -- > Best Regards > > ---------------------- > 刘明敏 | mmLiu >
