Correct, they will be run sequentially within the bolt. Threading within a bolt adds a ton of complication (needing to synchronize around output collector, for instance), so that's really bad. What are you trying to achieve by adding more threading? You won't get better resource usage because the rest of the topology is parallelized. You might get better latency, but in that case it would be better to just work on scheduler improvements to accomplish that and have the "split" tasks run as separate tasks within the same worker.
On Mon, Jan 27, 2014 at 5:14 AM, Simon Cooper < [email protected]> wrote: > I’ve been looking at the trident code. If the following code is run on > the same bolt: > > > > TridentStream input = … > > > > TridentStream firstSection = input.each(new Fields(…), new FirstFilter()); > > TridentStream secondSection = input.each(new Fields(…), new > SecondFilter()); > > > > From looking at FreshCollector and AppendCollector, it looks like the two > filters will be run sequentially. Are there plans/thoughts on involving > threading when a stream splits like this? Running each split on a new > thread in parallel? There may be some complications when joining streams > together, but that’s a simple solution that a couple of locks on the > MultiReducerProcessor would solve. Has this been considered before? > > > > SimonC > -- Twitter: @nathanmarz http://nathanmarz.com
