The majority of the problems we have seen have been due to i/o bottlenecks on the server. iostat will uncover these, though if you aren't familiar with it you may need to read through some docs. If iostat show high mb/sec, high number of operations (reads or writes) per second or high service time that is likely the issue.
-Jay On Tue, Jan 10, 2012 at 2:55 PM, Pierre-Yves Ritschard <p...@spootnik.org>wrote: > will try this, thanks > > On Tue, Jan 10, 2012 at 11:52 PM, Jun Rao <jun...@gmail.com> wrote: > > Pierre-Yves, > > > > I assume that this is a producer problem. If so, first, make sure flush > > interval is not too small. Second, you can enable debug logging in > > FileMessageSet to see log flush time. Then you can see if it's higher > than > > what you expect. > > > > Thanks, > > > > Jun > > > > > > On Tue, Jan 10, 2012 at 2:19 PM, Pierre-Yves Ritschard <p...@spootnik.org > >wrote: > > > >> Hi, > >> > >> One of my cluster runs on ec2 and I noticed that I have rather slow > >> throughput, less than 500 per second. I isolated the trouble on the > >> following setup > >> > >> 1 producer instance (m1.large) > >> 5 consumer instances also running a broker (m2.xlarge) > >> 1 zookeeper instance (m1.large) > >> > >> I am using a sync producer and have 60 partitions. > >> > >> On my local box, with everything running I would several thousand per > >> second. > >> >
