My point with increased throughput was that if you have items queued from the spout waiting to be processed, that counts towards the complete latency for the spout. If your bolts go through the tuples faster (and as you add more they do, you have 6x speedup from more bolts) then you will see the complete latency drop. On May 20, 2015 4:01 AM, "Dima Dragan" <[email protected]> wrote:
> Thank you, Jeffrey and Devang for your answers. > > Jeffrey, as far as I use shuffle grouping, I think, network serialization > will left, but there will be no network delays (for remove it there is > localOrShuffling grouping). For all experiments, I use only one worker, so > it does not explain why complete latency could decrease. > > But I think you are right about definitions) > > Devang, no, I set up 1 worker and 1 acker for all tests. > > > Best regards, > Dmytro Dragan > On May 20, 2015 05:03, "Devang Shah" <[email protected]> wrote: > >> Was the number of workers or number of ackers changed across your >> experiments ? What are the numbers you used ? >> >> When you have many executors, increasing the ackers reduces the complete >> latency. >> >> Thanks and Regards, >> Devang >> On 20 May 2015 03:15, "Jeffery Maass" <[email protected]> wrote: >> >>> Maybe the difference has to do with where the executors were running. >>> If your entire topology is running within the same worker, it would mean >>> that a serialization for the worker to worker networking layer is left out >>> of the picture. I suppose that would mean the complete latency could >>> decrease. At the same time, process latency could very well increase, >>> since all the work is being done within the same worker. My understanding >>> that process latency is measured from the time the tuple enters the >>> executor until it leaves the executor. Or was it from the time the tuple >>> enters the worker until it leaves the worker? I don't recall. >>> >>> I bet a firm definition of the latency terms would shed some light. >>> >>> Thank you for your time! >>> >>> +++++++++++++++++++++ >>> Jeff Maass <[email protected]> >>> linkedin.com/in/jeffmaass >>> stackoverflow.com/users/373418/maassql >>> +++++++++++++++++++++ >>> >>> >>> On Tue, May 19, 2015 at 9:47 AM, Dima Dragan <[email protected]> >>> wrote: >>> >>>> Thanks Nathan for your answer, >>>> >>>> But I`m afraid that you understand me wrong : With increasing >>>> executors by 32x, each executor's throughput *increased* by 5x, but >>>> complete latency dropped. >>>> >>>> On Tue, May 19, 2015 at 5:16 PM, Nathan Leung <[email protected]> >>>> wrote: >>>> >>>>> It depends on your application and the characteristics of the io. You >>>>> increased executors by 32x and each executor's throughput dropped by 5x, >>>>> so >>>>> it makes sense that latency will drop. >>>>> On May 19, 2015 9:54 AM, "Dima Dragan" <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> I have found a strange behavior in topology metrics. >>>>>> >>>>>> Let`s say, we have 1 node, 2-core machine. simple Storm topology >>>>>> Spout A -> Bolt B -> Bolt C >>>>>> >>>>>> Bolt B splits message on 320 parts and emits (shuffle grouping) each >>>>>> to Bolt C. Also Bolts B and C make some read/write operations to db. >>>>>> >>>>>> Input flow is continuous and static. >>>>>> >>>>>> Based on logic, setting up a more higher number of executors for Bolt >>>>>> C than number of cores should be useless (the bigger part of threads will >>>>>> be sleeping). >>>>>> It is confirmed by increasing execute and process latency. >>>>>> >>>>>> But I noticed that complete latency has started to decrease. And I do >>>>>> not understand why. >>>>>> >>>>>> For example, stats for bolt C: >>>>>> >>>>>> ExecutorsProcess latency (ms)Complete latency (ms)25.599897.27646.3 >>>>>> 526.36428.432345.454 >>>>>> >>>>>> Is it side effect of IO bound tasks? >>>>>> >>>>>> Thanks in advance. >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> Dmytro Dragan >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Dmytro Dragan >>>> >>>> >>>> >>>
