Hmmm, ok, I think that makes sense. So how would the buffering work? Can I just stash the tuple in memory and return without acking when I still need more? Then once I have all of them, process and ack them all at once? For some reason I thought it would be more complicated than that, but if it's that simple, great! :-)
Thanks! On Wed, Apr 29, 2015 at 6:15 AM Matthias J. Sax < [email protected]> wrote: > Hi, > > I am not familiar with CoordinatedBolt, but it seems not to do what you > want (it seem to do DRPC stuff) > > http://www.pixelmachine.org/storm/2012/01/03/How-CoordinatedBolt-Works.html > > To me, it seems as you would like to perform a simple join... For this, > you need to buffer all incoming meta-data tuples (that are related to > messages with attachments) in Dissem until the join is complete. For > this, you need to know (for each tuple coming from meta-data-transform) > how many attachment-tuples are expected from virus scanner. But Spout > can simple add this information. If the attachment-count-attribute is > zero, the message can be processed immediately. > > Does this make sense to you? > > However, I don't understand why you want to use direct-grouping? Using > fields-grouping on the message-id attribute should work for you. > > > -Matthias > > > On 04/29/2015 02:18 AM, Jason Kusar wrote: > > Hi, > > > > I'm currently working on building an ETL system using storm. > > Approximately 30% of the incoming records have binary attachments which > > need to be virus scanned. A single record can have one or more > > attachments. My initial thought was to build a topology with two outputs > > from the spout both of which eventually feed a downstream bolt. I've > > attached a simple diagram. Hopefully it comes through on the list. > > > > > > The spout would output tuples to the metadata transform on the default > > stream. If it came across a record that had attachments, it would output > > one or more additional tuples with the same ID to the Virus scan stream. > > Obviously the diagram is simplified as the Metadata transform might > > involve many steps, but regardless it's safe to assume that the time > > required for the virus scanner is likely significantly higher than the > > transform stream. I would like for records not having attachments to be > > able to keep flowing through the system without being slowed down by > > those records that do happen to have attachments. > > > > From looking at the CoordinatedBolt, it looks like it probably does > > exactly what I'm looking for, but I'm not sure. It would join the tuples > > from the two streams back together and deliver them to the dissem bolt > > as a batch to be processed from there. Am I viewing this right or am I > > completely off base? I can't find a lot of examples of CoordinatedBolts > > and there aren't any real comments in the code explaining what it's > doing. > > > > I feel like Direct Groupings might come into play here as well, but the > > link from the Documentation Manual page gets a 404, so I was unable to > > find more details on that. > > > > If I'm completely off, is there an example implementation that does > > something similar to what I'm trying to do? Specifically, is there an > > example of something outputting a variable number of tuples that all get > > grouped back together somewhere down the line? > > > > Thanks! > > --Jason > >
