Instead of doing sleep in spout, do conf.put(Config.TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS, 10); Otherwise there is another sleep(1) hidden in storm.
1. What can you tell about the spout code. Is it a mock that just performs a for loop with emit? 2. What does nimbus report the spout latency is ? On a side note, I would do benchmarks on a small Virtual Machine. Consider scaling up. Itai ________________________________________ From: Ahmed El Rheddane <[email protected]> Sent: Thursday, October 23, 2014 3:19 PM To: [email protected] Subject: Re: Generating a constant load of tuples Hi Itai, The bolt follows perfectly the pace of emission, I am using different workers for the components (each running on a seperate MS Azure small VM instance with 1 vCPU). CPU is nowhere near busy as I am sleeping for 10ms after each emission (confirmed by Ganglia monitoring). If it can help I'm using Strom 0.9.1 with Oracle Java 1.7. I also have the Zookeeper server and Nimbus on the same VM instance. Joined is the observed Spout throughput (should be a little less than 100 tuple/s at all time). Ahmed On 10/23/2014 01:19 PM, Itai Frenkel wrote: > I meant 100% CPU on any of the CPU cores? > ________________________________________ > From: Itai Frenkel<[email protected]> > Sent: Thursday, October 23, 2014 2:11 PM > To:[email protected] > Subject: Re: Generating a constant load of tuples > > Conjecture - Could the bolt after the spout be the problem? Once the bolt > input queue is full (I think the default is 1024 items), then it will > pushback and Spout emit will block the spout thread. > You could measure the time it took to emit for diagnostics. > > Also- do you have 2 CPUs. > > Itai > ________________________________________ > From: Ahmed El Rheddane<[email protected]> > Sent: Thursday, October 23, 2014 11:52 AM > To:[email protected] > Subject: Generating a constant load of tuples > > Hello again, > > I am trying to generate a regulated load of tuples using a sleep in the > spout's nextTuple(). However, the observed results show that the number > of tuples periodically drops by about 200 hundred tuples (from 800 to 600). > > I thought that this was due to the way I was monitoring Storm, but the > behvior has been confirmed by local spout logs. > I haven't set any limit on pending messages and I'm not using any ackers > anyways. > > Any help would be very much appreciated. > > Cheers, > Ahmed
