That's true. Group by ID field and hold the processed tuples in a RotatingMap at final bolt till all are received. Rotate the map with tick tuples to expire unused tuples.
On Fri, Aug 1, 2014 at 10:14 AM, Varun Vijayaraghavan <[email protected]> wrote: > That's interesting. Note that I have not used such a pattern before - and > have done something similar. I have not used trident - so this probably > will not answer your last question completely :) > > If you set up the topology such that links between bolt {1, 2, 3} and > final bolt is stream grouped by "msgId" - you could keep the partially > processed results in memory (or in a persisted state somewhere) - till you > see the processed result for all the bolts. > > I would also expire msgIds which have not seen further results for beyond > a certain threshold time. > > What do you think? > > > > On Fri, Aug 1, 2014 at 12:35 AM, Spico Florin <[email protected]> > wrote: > >> Hello! >> I have a case study where the same message (identified by an id) is >> spread over a couple of processing bolts and a final bolt should act as a >> barrier. This final bolt should do its job on the message ID only when all >> the upfront bolts have finished their process on the message. >> >> As sketch is bellow >> >> >> Spout -> (msgid1, payload)->process bolt 1 (msgId1, payloadb1) | >> ->process bolt 2(msgId1, >> payloadb2) |->final bolt(msg1,finalp) >> ->process bolt 3(msgId1, >> payloadb3) | >> >> So the final bolt should not start a work on a message id till the >> message was not processed by all the 3 processing bolts. >> My question are: >> 1. Can be these case viable for storm? >> 2. If the answer for the first question is yes, how can I achieve this >> request? >> 3. Is possible to achieve this request without using trident? >> I look forward for your answers. >> Thanks, >> Florin >> > > > > -- > - varun :) >
