Generally yes, the best case for output collector is passing a reference through some queues. However, it's harder to reason about the performance of a larger topology, and (assuming you use reliable messaging) your entire topology can be held up by one poorly performing bolt. On Nov 5, 2015 7:31 AM, "Crina Arsenie" <[email protected]> wrote:
> Hello, > > I'm interested in this topic also. Thank you for your answer. > I didn't knew about Flux, maybe it could do the job for my case, i'll take > a lot at it. > I have a also question about performance, I assume that passing through > the output collector is faster than Kaka, what do you think ? > > Thank you, > > Crina > > 2015-11-05 12:36 GMT+01:00 Nathan Leung <[email protected]>: > >> It's not possible to combine several topologies into one, but it should >> be possible to write different tuple sinks such that you can configure each >> bolt to write to either the output collector or Kafka. Then it's just a >> matter of wiring and configuring your bolts differently. >> >> You can use something like flux ( >> http://storm.apache.org/documentation/flux.html) to change how your >> bolts are wired without having to rebuild your jar file every time. >> On Nov 5, 2015 4:17 AM, "Irina Alles" <[email protected]> wrote: >> >>> Hello, >>> >>> >>> >>> We are currently studying if storm would be appropriate as a part of our >>> monitoring system. >>> >>> We are measuring sensor data, we need to apply different transformation >>> steps and store it somewhere or send it further. >>> >>> This seems to be a basic use case for storm so far, let’s call this >>> setup topology A. >>> >>> >>> We would like to be able to add measured sensors and their >>> transformation steps (topology B) dynamically without requiring any system >>> downtime. These dynamic additions could happen frequently. It will happen >>> that bolts of topology B will require the output of certain bolts in A >>> >>> Is there a best practice to manage this in storm? >>> >>> >>> >>> After a certain runtime of the system we will have to manage several >>> topologies and we need to assure the communication between them. >>> >>> We thought about using Kafka for the communication between topologies, >>> but with the growing number of topologies this might not be the best >>> approach I suppose (‘best’ means in this case: easy to handle, avoiding >>> message overhead). >>> >>> Maybe it would be better to group some topologies to create a greater >>> one? How would one do this in storm (I’ve read about the swap feature, but >>> it doesn’t seem to be available yet)? >>> >>> Is there a better approach? >>> >>> >>> >>> Thanks! >>> >>> Irina >>> >> > >
