It's not possible to combine several topologies into one, but it should be possible to write different tuple sinks such that you can configure each bolt to write to either the output collector or Kafka. Then it's just a matter of wiring and configuring your bolts differently.
You can use something like flux ( http://storm.apache.org/documentation/flux.html) to change how your bolts are wired without having to rebuild your jar file every time. On Nov 5, 2015 4:17 AM, "Irina Alles" <[email protected]> wrote: > Hello, > > > > We are currently studying if storm would be appropriate as a part of our > monitoring system. > > We are measuring sensor data, we need to apply different transformation > steps and store it somewhere or send it further. > > This seems to be a basic use case for storm so far, let’s call this setup > topology A. > > > We would like to be able to add measured sensors and their transformation > steps (topology B) dynamically without requiring any system downtime. These > dynamic additions could happen frequently. It will happen that bolts of > topology B will require the output of certain bolts in A > > Is there a best practice to manage this in storm? > > > > After a certain runtime of the system we will have to manage several > topologies and we need to assure the communication between them. > > We thought about using Kafka for the communication between topologies, but > with the growing number of topologies this might not be the best approach I > suppose (‘best’ means in this case: easy to handle, avoiding message > overhead). > > Maybe it would be better to group some topologies to create a greater one? > How would one do this in storm (I’ve read about the swap feature, but it > doesn’t seem to be available yet)? > > Is there a better approach? > > > > Thanks! > > Irina >
