Whether timeouts are same, I hope someone else would be able to confirm. I haven't worked with Trident extensively. If you wanted your REST calls parallelized, you could also design it to have a separate program or topology write data into a queue and you could have multiple spouts reading from that queue and then the REST calls could be done in the spout itself. Think differently to solve design issues.
On Mon, Aug 22, 2016 at 2:52 PM, Pratyusha Rasamsetty < [email protected]> wrote: > Timouts problem can be same in both storm and trident? > > Topologies can be as big as you want them to, but I have personally >> preferred small ones because of the same problem that you are facing. >> Timeouts. >> For you, I'd recommend that after your trident state inserts to DB, send >> an ack and don't do any further emits. >> Let the lower part of your diagram be a separate topology that queries >> what Trident wrote and processes it. >> > In Trident state, I am maintaining some data in > > memory as well - basically the aggregated data of around 100 tuples > emitted by previous bolt, to know whether the processing is done or not. > This information (processed tuples) will be used for further processing > >> Also, each REST call section and the "each" bolt before it could be moved >> into the spout. >> > If I move those to spout, won't I be missing all the parallelism required > to call the REST API's > > > Please do let me know. > > -- Regards, Navin
