So, you want to queue up the data awaiting a key match on the enrichment data, up to a max timeout and/or buffer size? Seems like this should belong at the spout level to avoid buffer overflows, depending on how big the data sets are and how far apart the matching records/elements are spaced in time.
On Fri, Nov 4, 2016 at 7:28 AM, [email protected] <[email protected]> wrote: > Is there a good method (i.e. something using Stellar/ZK) to implement an > intentional processing delay to all tuples in a specific topology? I plan > to do some custom enrichments, but the data used to do the enrichment > *may* be > ingested at roughly the same time the data to be enriched is (it also may > not ever be sent). So I'd like to add a delay in my cluster that applies > to certain parser topologies. > > I took a look around in the documentation and in JIRA and didn't find > anything available or being worked on, but I did see that this may conflict > with METRON-322. Essentially what I'm considering is a {sleep,delay,wait} > stellar function, but it could also be a delay in a parser's kafka spout > (much less of a fan of the second option). > > I'm looking for feedback on the best way to approach this, and I'd be happy > to do the work myself (if necessary) when it gets to that point. I did > consider implementing this delay upstream (in the sensor itself), but after > looking in more detail it doesn't seem as feasible. > > Jon > -- > > Jon >
