How about merging topologies into one? Though tuple timeout should be set to max processing time into all of topologies, there's only way to work without adding other components.
Btw, ideally supporting pub-sub between topology seems great, but AFAIK there're many hurdles to realize. 1. Subscribing spout should replay tuple when failure occurs (by Guarantee message processing), but publishing bolt can't help to do it. 2. Spout should have feature to receive from bolt (by TCP), which isn't exist yet. 3. Spout retrieves data from data source when nextTuple() occurs, which may can't applied to pub-sub situation. 4. Pub-sub spouts/bolts should allow task registration dynamically (maybe it's already exist) So I also recommend adding message queue(kafka, rabbitmq, etc.) between topologies. Please correct me if I'm wrong. Regards. Jungtaek Lim (HeartSaVioR) 2014년 10월 17일 금요일, Klausen Schaefersinho<[email protected]>님이 작성한 메시지: > Hi, > > In my storm setup data arrives in form of files that I have to read and > emit in my spout. Also my topology is very dynamic. Some topologies run > quite long, whereas other can turned on and off frequently. In order to > avoid that I have n spouts reading from the files, I was wondering if could > have just one topology in the cluster which reads from the file and just > emits tuples? All other topologies would than register and that "listen" to > taht topology. > > Cheers, > > Klaus > -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
