Thank you a lot for the fast replies! Flux sounds good, we will have a deeper look into how we could apply it.
Regards, Irina On 6 November 2015 at 10:33, Crina Arsenie <[email protected]> wrote: > Thank you all for your inputs. @santosh I think we will continue using > Storm for the moment. And as I saw, Samza does almost the same but in a > different way and from their site is even written that "Samza is pretty > immature. " @saiprasad maybe we can try using groovy scripts yes! > > Thanks again, > Crina > > > 2015-11-05 17:39 GMT+01:00 saiprasad mishra <[email protected]>: > >> You can explore creating a new groovy script for transformation every >> time and push it to a predefined location in all worker nodes and use it in >> a bolt to execute it based on some tuple value field. Spring can help >> execute the groovy script as bean too. >> >> Regards >> Sai >> >> On Thu, Nov 5, 2015 at 7:05 AM, Santosh Pingale <[email protected] >> > wrote: >> >>> Apache Samza fits the requirements. You can give it a shot. But whatever >>> Nathan said can still cause a big toll on overall performance but that >>> depends on your DAG. >>> On Nov 5, 2015 6:25 PM, "Nathan Leung" <[email protected]> wrote: >>> >>>> Generally yes, the best case for output collector is passing a >>>> reference through some queues. However, it's harder to reason about the >>>> performance of a larger topology, and (assuming you use reliable messaging) >>>> your entire topology can be held up by one poorly performing bolt. >>>> On Nov 5, 2015 7:31 AM, "Crina Arsenie" <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm interested in this topic also. Thank you for your answer. >>>>> I didn't knew about Flux, maybe it could do the job for my case, i'll >>>>> take a lot at it. >>>>> I have a also question about performance, I assume that passing >>>>> through the output collector is faster than Kaka, what do you think ? >>>>> >>>>> Thank you, >>>>> >>>>> Crina >>>>> >>>>> 2015-11-05 12:36 GMT+01:00 Nathan Leung <[email protected]>: >>>>> >>>>>> It's not possible to combine several topologies into one, but it >>>>>> should be possible to write different tuple sinks such that you can >>>>>> configure each bolt to write to either the output collector or Kafka. >>>>>> Then >>>>>> it's just a matter of wiring and configuring your bolts differently. >>>>>> >>>>>> You can use something like flux ( >>>>>> http://storm.apache.org/documentation/flux.html) to change how your >>>>>> bolts are wired without having to rebuild your jar file every time. >>>>>> On Nov 5, 2015 4:17 AM, "Irina Alles" <[email protected]> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> >>>>>>> >>>>>>> We are currently studying if storm would be appropriate as a part of >>>>>>> our monitoring system. >>>>>>> >>>>>>> We are measuring sensor data, we need to apply different >>>>>>> transformation steps and store it somewhere or send it further. >>>>>>> >>>>>>> This seems to be a basic use case for storm so far, let’s call this >>>>>>> setup topology A. >>>>>>> >>>>>>> >>>>>>> We would like to be able to add measured sensors and their >>>>>>> transformation steps (topology B) dynamically without requiring any >>>>>>> system >>>>>>> downtime. These dynamic additions could happen frequently. It will >>>>>>> happen >>>>>>> that bolts of topology B will require the output of certain bolts in A >>>>>>> >>>>>>> Is there a best practice to manage this in storm? >>>>>>> >>>>>>> >>>>>>> >>>>>>> After a certain runtime of the system we will have to manage several >>>>>>> topologies and we need to assure the communication between them. >>>>>>> >>>>>>> We thought about using Kafka for the communication between >>>>>>> topologies, but with the growing number of topologies this might not be >>>>>>> the >>>>>>> best approach I suppose (‘best’ means in this case: easy to handle, >>>>>>> avoiding message overhead). >>>>>>> >>>>>>> Maybe it would be better to group some topologies to create a >>>>>>> greater one? How would one do this in storm (I’ve read about the swap >>>>>>> feature, but it doesn’t seem to be available yet)? >>>>>>> >>>>>>> Is there a better approach? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Irina >>>>>>> >>>>>> >>>>> >>>>> >> > >
