On Wed, Mar 19, 2014 at 1:28 PM, Eugene Dzhurinsky <[email protected]> wrote: > > On Wed, Mar 19, 2014 at 01:03:43PM -0700, Angelo Genovese wrote: > > "Any problem in computer science can be solved with another layer of > > indirection." (David Wheeler) > > ... and every new layer of indirection introduces new bugs, which will combine > with the existings ones and produce up to N*M new problems, where N is number > of bugs in the lower layer, and M - in the upper one. (Unknown Developer)
The quote was meant to be a bit tongue in cheek, sorry. > > You could setup the spout to read it's list of sources from somewhere, > > add > > to that list when a request comes in, then remove completed sources from > > that list in your sink. Or you could use a separate app to read from the > > sources on request and pile that info into a queue like Kestrel for a > > storm spout to read from. > > Yes, that will probably solve the task of adding new data sources, however > it's still not clear how to mark some task as "complete", so the sink could > send the results back to the user. Like if the spout had produced 5 messages > and 6-th is "EOT", and that EOT message arrived to sink when the message 3 was > in progress. > >From the sounds of it you need some sort of coordination, which from what I've seen so far is only really possible with a single starting tuple. If that's the case (I'm still pretty new to storm myself, I've only been working on it for a few months now), you could have a spout emit the new external source and build a bolt to connect to it and emit the relevant chunks. you would need to coordinate that and return your response to the user in the finished callback for your sink. This is a bit similar to something we're working on where we have a url coming in on a kestrel queue and bolts which fetch the content of the page, break it up and process the pieces, then a bolt which accumulates the results of that processing and writes the aggregate information to a data-store in its finished callback. Ciao, -- Angelo Genovese
